Nov 02 17:30:34.064194 ip-10-0-141-169 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 17:30:34.064204 ip-10-0-141-169 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 17:30:34.064211 ip-10-0-141-169 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 17:30:34.064462 ip-10-0-141-169 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 17:30:44.079987 ip-10-0-141-169 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 17:30:44.080005 ip-10-0-141-169 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7a7ca67dc6344e85a3106a573ae23c48 -- Nov 02 17:32:44.649935 ip-10-0-141-169 systemd[1]: Starting Kubernetes Kubelet... Nov 02 17:32:45.107862 ip-10-0-141-169 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:32:45.107862 ip-10-0-141-169 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 17:32:45.107862 ip-10-0-141-169 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:32:45.107862 ip-10-0-141-169 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:32:45.112042 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.111949 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:32:45.112042 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112042 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112046 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112049 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112052 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112055 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112057 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112060 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112063 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112066 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112069 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112071 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112076 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112082 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112085 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112095 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112098 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112101 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112103 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112106 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112109 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:32:45.112116 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112111 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112114 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112117 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112120 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112123 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112126 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112129 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112131 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112134 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112137 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112139 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112142 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112144 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112147 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112149 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112152 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112154 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112157 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112160 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112163 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:32:45.112577 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112165 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:32:45.113120 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112168 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:32:45.113120 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112170 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:32:45.113120 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112173 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:32:45.113120 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112175 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:32:45.113120 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112178 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:32:45.113120 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112180 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:32:45.113120 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112183 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:32:45.113120 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112185 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:32:45.113120 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112188 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:32:45.113120 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112190 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:32:45.113120 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112193 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:32:45.113120 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112196 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:32:45.113120 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112199 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:32:45.113120 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112203 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:32:45.113120 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112205 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:32:45.113120 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112208 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:32:45.113120 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112211 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:32:45.113120 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112215 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:32:45.113120 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112219 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112222 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112225 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112228 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112230 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112233 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112235 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112238 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112240 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112243 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112245 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112248 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112251 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112254 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112257 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112260 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112263 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112265 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112268 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112271 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:32:45.113572 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112273 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112276 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112279 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112281 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112284 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112287 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112290 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112292 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112295 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112298 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112300 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112303 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112305 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112308 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112310 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112313 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112315 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112318 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112320 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112323 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:32:45.114062 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112325 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:32:45.114555 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112328 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:32:45.114555 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112330 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:32:45.114555 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112333 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:32:45.114555 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112336 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:32:45.114555 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.112339 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:32:45.114555 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113131 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:32:45.114555 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113138 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:32:45.114555 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113141 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:32:45.114555 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113145 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:32:45.114555 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113148 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:32:45.114555 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113151 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:32:45.114555 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113153 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:32:45.114555 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113157 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:32:45.114555 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113159 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:32:45.114555 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113162 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:32:45.114555 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113165 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:32:45.114555 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113167 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:32:45.114555 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113170 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:32:45.114555 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113172 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:32:45.115023 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113175 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:32:45.115023 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113178 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:32:45.115023 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113181 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:32:45.115023 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113183 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:32:45.115023 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113186 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:32:45.115023 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113189 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:32:45.115023 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113191 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:32:45.115023 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113194 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:32:45.115023 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113197 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:32:45.115023 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113200 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:32:45.115023 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113202 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:32:45.115023 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113205 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:32:45.115023 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113208 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:32:45.115023 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113211 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:32:45.115023 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113213 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:32:45.115023 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113216 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:32:45.115023 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113218 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:32:45.115023 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113221 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:32:45.115466 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113229 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:32:45.115466 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113234 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:32:45.115466 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113239 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:32:45.115466 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113242 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:32:45.115466 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113245 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:32:45.115466 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113248 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:32:45.115466 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113252 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:32:45.115466 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113254 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:32:45.115466 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113257 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:32:45.115466 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113259 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:32:45.115466 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113262 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:32:45.115466 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113265 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:32:45.115466 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113267 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:32:45.115466 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113270 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:32:45.115466 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113272 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:32:45.115466 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113275 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:32:45.115466 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113279 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:32:45.115466 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113282 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:32:45.115466 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113285 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113287 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113290 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113293 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113295 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113297 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113300 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113302 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113305 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113308 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113310 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113313 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113316 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113318 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113321 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113324 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113326 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113329 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113332 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113335 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:32:45.115924 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113337 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113340 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113342 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113345 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113347 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113350 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113352 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113356 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113360 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113363 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113366 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113368 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113371 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113373 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113376 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113378 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113381 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113384 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113386 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113389 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:32:45.116403 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113391 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113394 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113397 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113399 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113402 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113404 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113407 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113410 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113412 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113415 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113421 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113424 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113427 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113430 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113433 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113435 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113969 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113976 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113979 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113982 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:32:45.116889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113985 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113988 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113991 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113994 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113996 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.113999 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114002 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114005 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114007 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114010 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114013 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114015 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114018 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114020 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114023 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114026 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114028 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114031 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114034 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114036 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:32:45.117356 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114039 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114042 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114047 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114050 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114052 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114055 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114058 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114060 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114063 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114066 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114069 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114072 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114075 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114077 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114080 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114083 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114086 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114089 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114091 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114094 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:32:45.117868 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114096 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114099 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114101 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114104 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114107 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114109 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114112 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114114 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114117 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114120 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114122 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114125 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114127 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114131 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114134 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114139 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114141 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114144 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114146 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114149 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:32:45.118337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114151 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:32:45.118822 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114154 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:32:45.118822 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114156 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:32:45.118822 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114159 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:32:45.118822 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114162 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:32:45.118822 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114164 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:32:45.118822 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114167 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:32:45.118822 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114169 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:32:45.118822 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114172 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:32:45.118822 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114174 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:32:45.118822 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114177 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:32:45.118822 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114179 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:32:45.118822 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114182 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:32:45.118822 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114184 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:32:45.118822 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114187 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:32:45.118822 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114189 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:32:45.118822 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114193 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:32:45.118822 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114197 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:32:45.118822 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114200 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:32:45.118822 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114203 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:32:45.119297 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114205 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:32:45.119297 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114208 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:32:45.119297 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114211 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:32:45.119297 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114213 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:32:45.119297 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114216 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:32:45.119297 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114218 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:32:45.119297 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114221 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:32:45.119297 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114225 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:32:45.119297 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114228 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:32:45.119297 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114231 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:32:45.119297 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114234 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:32:45.119297 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114236 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:32:45.119297 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114239 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:32:45.119297 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114241 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:32:45.119297 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114244 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:32:45.119297 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114246 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:32:45.119297 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114249 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:32:45.119297 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114251 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:32:45.119297 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114256 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114260 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114263 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.114265 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115902 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115914 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115922 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115927 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115932 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115936 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115940 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115945 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115949 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115952 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115955 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115959 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115962 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115965 2579 flags.go:64] FLAG: --cgroup-root="" Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115968 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115972 2579 flags.go:64] FLAG: --client-ca-file="" Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115975 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115979 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115983 2579 flags.go:64] FLAG: --cluster-domain="" Nov 02 17:32:45.119809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115987 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115990 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115993 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.115996 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116000 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116004 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116007 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116010 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116014 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116017 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116020 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116023 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116026 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116030 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116034 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116037 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116040 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116043 2579 flags.go:64] FLAG: --enable-server="true" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116046 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116051 2579 flags.go:64] FLAG: --event-burst="100" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116054 2579 flags.go:64] FLAG: --event-qps="50" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116057 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116060 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116063 2579 flags.go:64] FLAG: --eviction-hard="" Nov 02 17:32:45.120347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116068 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116071 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116074 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116077 2579 flags.go:64] FLAG: --eviction-soft="" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116080 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116083 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116086 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116090 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116093 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116096 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116099 2579 flags.go:64] FLAG: --feature-gates="" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116103 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116106 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116109 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116112 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116116 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116119 2579 flags.go:64] FLAG: --help="false" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116122 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116126 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116129 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116132 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116135 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116138 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 17:32:45.120985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116142 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116145 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116148 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116151 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116154 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116157 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116160 2579 flags.go:64] FLAG: --kube-reserved="" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116163 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116167 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116171 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116174 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116177 2579 flags.go:64] FLAG: --lock-file="" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116180 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116183 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116186 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116192 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116195 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116199 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116202 2579 flags.go:64] FLAG: --logging-format="text" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116205 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116208 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116211 2579 flags.go:64] FLAG: --manifest-url="" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116214 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116219 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116222 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 17:32:45.121530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116226 2579 flags.go:64] FLAG: --max-pods="110" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116230 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116233 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116236 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116239 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116242 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116245 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116249 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116256 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116259 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116262 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116265 2579 flags.go:64] FLAG: --pod-cidr="" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116268 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116271 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116274 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116279 2579 flags.go:64] FLAG: --port="10250" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116282 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116285 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-03667256e140917f5" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116288 2579 flags.go:64] FLAG: --qos-reserved="" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116291 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116294 2579 flags.go:64] FLAG: --register-node="true" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116297 2579 flags.go:64] FLAG: --register-with-taints="" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116302 2579 flags.go:64] FLAG: --registry-burst="10" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116305 2579 flags.go:64] FLAG: --registry-qps="5" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116308 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116312 2579 flags.go:64] FLAG: --reserved-memory="" Nov 02 17:32:45.122153 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116316 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116319 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116322 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116325 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116328 2579 flags.go:64] FLAG: --runonce="false" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116331 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116335 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116339 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116341 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116344 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116348 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116351 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116355 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116357 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116361 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116366 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116369 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116372 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116375 2579 flags.go:64] FLAG: --system-cgroups="" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116378 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116384 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116389 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116392 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116396 2579 flags.go:64] FLAG: --tls-min-version="" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116399 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 17:32:45.122861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116402 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116405 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116408 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116412 2579 flags.go:64] FLAG: --v="2" Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116416 2579 flags.go:64] FLAG: --version="false" Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116421 2579 flags.go:64] FLAG: --vmodule="" Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116425 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116429 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116571 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116575 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116578 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116581 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116584 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116587 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116590 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116592 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116595 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116598 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116600 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116603 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116605 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:32:45.123455 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116608 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116625 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116631 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116635 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116637 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116640 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116643 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116647 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116650 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116652 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116655 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116658 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116660 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116663 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116666 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116668 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116671 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116673 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116676 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116679 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:32:45.124005 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116682 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:32:45.124474 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116685 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:32:45.124474 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116687 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:32:45.124474 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116690 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:32:45.124474 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116693 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:32:45.124474 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116695 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:32:45.124474 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116698 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:32:45.124474 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116701 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:32:45.124474 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116704 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:32:45.124474 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116706 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:32:45.124474 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116709 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:32:45.124474 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116711 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:32:45.124474 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116714 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:32:45.124474 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116718 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:32:45.124474 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116721 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:32:45.124474 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116723 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:32:45.124474 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116726 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:32:45.124474 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116729 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:32:45.124474 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116731 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:32:45.124474 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116735 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116738 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116742 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116746 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116749 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116752 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116755 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116757 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116760 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116763 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116766 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116768 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116771 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116773 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116776 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116779 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116782 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116784 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116787 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116789 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:32:45.124938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116792 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:32:45.125406 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116795 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:32:45.125406 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116797 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:32:45.125406 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116800 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:32:45.125406 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116802 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:32:45.125406 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116805 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:32:45.125406 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116809 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:32:45.125406 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116811 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:32:45.125406 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116814 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:32:45.125406 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116816 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:32:45.125406 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116819 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:32:45.125406 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116822 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:32:45.125406 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116825 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:32:45.125406 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116828 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:32:45.125406 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116831 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:32:45.125406 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116833 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:32:45.125406 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116836 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:32:45.125406 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116839 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:32:45.125406 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116843 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:32:45.125406 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116846 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:32:45.125891 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116850 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:32:45.125891 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116852 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:32:45.125891 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116855 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:32:45.125891 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116858 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:32:45.125891 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116860 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:32:45.125891 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116863 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:32:45.125891 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116866 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:32:45.125891 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116869 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:32:45.125891 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116871 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:32:45.125891 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116874 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:32:45.125891 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116876 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:32:45.125891 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116879 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:32:45.125891 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116881 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:32:45.125891 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116884 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:32:45.125891 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.116912 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:32:45.125891 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116952 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:32:45.126273 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116956 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:32:45.126273 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116959 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:32:45.126273 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116963 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:32:45.126273 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116966 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:32:45.126273 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116968 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:32:45.126273 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116971 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:32:45.126273 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116974 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:32:45.126273 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116976 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:32:45.126273 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116979 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:32:45.126273 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116983 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:32:45.126273 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116985 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:32:45.126273 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116988 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:32:45.126273 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116991 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:32:45.126273 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116996 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:32:45.126273 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.116999 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:32:45.126273 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117003 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:32:45.126273 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117006 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:32:45.126273 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117009 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:32:45.126273 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117011 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117014 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117017 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117020 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117022 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117025 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117028 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117031 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117034 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117036 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117039 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117042 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117044 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117046 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117049 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117052 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117056 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117058 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117061 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117064 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:32:45.126733 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117066 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117069 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117071 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117074 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117077 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117079 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117082 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117085 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117087 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117090 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117094 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117099 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117102 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117105 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117108 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117110 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117113 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117115 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117120 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117122 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:32:45.127213 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117125 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:32:45.127719 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117127 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:32:45.127719 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117130 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:32:45.127719 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117132 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:32:45.127719 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117135 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:32:45.127719 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117137 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:32:45.127719 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117140 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:32:45.127719 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117143 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:32:45.127719 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117147 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:32:45.127719 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117150 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:32:45.127719 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117152 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:32:45.127719 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117155 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:32:45.127719 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117157 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:32:45.127719 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117160 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:32:45.127719 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117162 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:32:45.127719 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117165 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:32:45.127719 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117168 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:32:45.127719 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117170 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:32:45.127719 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117173 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:32:45.127719 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117175 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117178 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117180 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117183 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117186 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117188 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117191 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117193 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117196 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117199 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117201 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117204 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117207 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117209 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117212 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117215 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117217 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117219 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117222 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117224 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:32:45.128177 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117227 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:32:45.128725 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117231 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:32:45.128725 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117234 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:32:45.128725 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117237 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:32:45.128725 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117240 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:32:45.128725 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117242 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:32:45.128725 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117245 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:32:45.128725 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.117247 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:32:45.128725 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.124598 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 17:32:45.128725 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.124629 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 17:32:45.128725 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.124646 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 17:32:45.128725 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.124653 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 17:32:45.128725 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124725 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:32:45.128725 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124734 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:32:45.128725 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124738 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:32:45.128725 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124743 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:32:45.128725 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124748 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:32:45.128725 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124752 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:32:45.128725 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124756 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:32:45.128725 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124761 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:32:45.129165 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124765 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:32:45.129165 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124770 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:32:45.129165 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124774 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:32:45.129165 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124778 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:32:45.129165 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124784 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:32:45.129165 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124788 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:32:45.129165 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124792 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:32:45.129165 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124796 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:32:45.129165 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124801 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:32:45.129165 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124806 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:32:45.129165 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124810 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:32:45.129165 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124814 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:32:45.129165 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124818 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:32:45.129165 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124823 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:32:45.129165 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124827 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:32:45.129165 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124831 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:32:45.129165 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124835 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:32:45.129165 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124839 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:32:45.129165 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124843 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124848 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124852 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124857 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124862 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124866 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124870 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124875 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124879 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124883 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124887 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124891 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124895 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124899 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124903 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124907 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124911 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124915 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124919 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124924 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:32:45.129692 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124928 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:32:45.130160 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124934 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:32:45.130160 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124941 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:32:45.130160 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124946 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:32:45.130160 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124950 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:32:45.130160 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124955 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:32:45.130160 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124959 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:32:45.130160 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124963 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:32:45.130160 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124967 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:32:45.130160 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124971 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:32:45.130160 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124975 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:32:45.130160 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124979 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:32:45.130160 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124983 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:32:45.130160 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124987 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:32:45.130160 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124991 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:32:45.130160 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.124996 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:32:45.130160 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125000 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:32:45.130160 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125004 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:32:45.130160 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125009 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:32:45.130160 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125013 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:32:45.130591 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125017 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:32:45.130591 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125021 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:32:45.130591 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125025 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:32:45.130591 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125029 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:32:45.130591 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125033 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:32:45.130591 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125038 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:32:45.130591 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125042 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:32:45.130591 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125046 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:32:45.130591 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125050 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:32:45.130591 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125054 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:32:45.130591 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125059 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:32:45.130591 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125063 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:32:45.130591 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125068 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:32:45.130591 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125073 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:32:45.130591 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125077 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:32:45.130591 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125081 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:32:45.130591 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125085 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:32:45.130591 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125090 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:32:45.130591 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125095 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:32:45.131046 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125100 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:32:45.131046 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125105 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:32:45.131046 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125108 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:32:45.131046 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125113 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:32:45.131046 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125117 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:32:45.131046 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125121 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:32:45.131046 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125125 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:32:45.131046 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125129 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:32:45.131046 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125133 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:32:45.131046 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125138 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:32:45.131046 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125142 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:32:45.131046 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125146 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:32:45.131046 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125150 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:32:45.131046 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125155 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:32:45.131046 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125159 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:32:45.131046 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125163 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:32:45.131046 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125167 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:32:45.131046 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125172 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:32:45.131046 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125176 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:32:45.131490 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125180 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:32:45.131490 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125184 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:32:45.131490 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.125221 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:32:45.131490 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125398 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:32:45.131490 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125406 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:32:45.131490 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125411 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:32:45.131490 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125416 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:32:45.131490 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125421 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:32:45.131490 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125425 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:32:45.131490 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125430 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:32:45.131490 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125434 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:32:45.131490 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125438 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:32:45.131490 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125442 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:32:45.131490 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125446 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:32:45.131490 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125450 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:32:45.131490 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125454 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:32:45.131889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125458 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:32:45.131889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125462 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:32:45.131889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125466 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:32:45.131889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125470 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:32:45.131889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125475 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:32:45.131889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125480 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:32:45.131889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125485 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:32:45.131889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125489 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:32:45.131889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125494 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:32:45.131889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125498 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:32:45.131889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125501 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:32:45.131889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125506 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:32:45.131889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125510 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:32:45.131889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125514 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:32:45.131889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125518 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:32:45.131889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125523 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:32:45.131889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125527 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:32:45.131889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125531 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:32:45.131889 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125535 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125539 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125544 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125548 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125552 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125556 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125561 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125565 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125569 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125573 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125578 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125582 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125586 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125591 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125595 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125599 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125602 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125607 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125611 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125628 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:32:45.132340 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125632 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:32:45.132873 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125636 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:32:45.132873 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125640 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:32:45.132873 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125644 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:32:45.132873 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125650 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:32:45.132873 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125654 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:32:45.132873 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125658 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:32:45.132873 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125662 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:32:45.132873 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125668 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:32:45.132873 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125673 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:32:45.132873 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125678 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:32:45.132873 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125682 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:32:45.132873 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125686 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:32:45.132873 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125690 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:32:45.132873 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125695 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:32:45.132873 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125699 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:32:45.132873 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125704 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:32:45.132873 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125708 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:32:45.132873 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125712 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:32:45.132873 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125717 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:32:45.133318 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125722 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:32:45.133318 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125726 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:32:45.133318 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125730 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:32:45.133318 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125734 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:32:45.133318 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125738 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:32:45.133318 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125742 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:32:45.133318 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125746 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:32:45.133318 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125750 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:32:45.133318 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125754 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:32:45.133318 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125758 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:32:45.133318 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125763 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:32:45.133318 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125767 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:32:45.133318 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125771 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:32:45.133318 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125775 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:32:45.133318 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125779 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:32:45.133318 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125784 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:32:45.133318 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125788 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:32:45.133318 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125793 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:32:45.133318 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125797 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:32:45.133938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125801 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:32:45.133938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125805 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:32:45.133938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125809 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:32:45.133938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125814 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:32:45.133938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125818 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:32:45.133938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125822 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:32:45.133938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125826 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:32:45.133938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125829 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:32:45.133938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125834 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:32:45.133938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125839 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:32:45.133938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125845 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:32:45.133938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125849 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:32:45.133938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125853 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:32:45.133938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125859 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:32:45.133938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125863 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:32:45.133938 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125867 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:32:45.134539 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.125903 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:32:45.134539 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125947 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:32:45.134539 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125953 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:32:45.134539 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125957 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:32:45.134539 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125961 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:32:45.134539 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125965 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:32:45.134539 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125970 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:32:45.134539 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125975 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:32:45.134539 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125980 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:32:45.134539 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125984 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:32:45.134539 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125989 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:32:45.134539 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125993 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:32:45.134539 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.125997 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:32:45.134539 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126001 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:32:45.134539 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126005 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126009 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126013 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126017 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126021 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126025 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126029 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126033 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126037 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126040 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126045 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126049 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126053 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126058 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126062 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126066 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126071 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126075 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126081 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126085 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:32:45.135021 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126091 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126097 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126101 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126106 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126110 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126114 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126118 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126122 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126126 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126130 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126134 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126138 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126142 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126146 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126150 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126154 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126159 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126163 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126166 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126170 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:32:45.135510 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126175 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:32:45.135995 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126179 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:32:45.135995 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126183 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:32:45.135995 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126187 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:32:45.135995 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126192 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:32:45.135995 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126196 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:32:45.135995 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126200 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:32:45.135995 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126205 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:32:45.135995 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126210 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:32:45.135995 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126214 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:32:45.135995 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126220 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:32:45.135995 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126223 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:32:45.135995 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126228 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:32:45.135995 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126232 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:32:45.135995 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126235 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:32:45.135995 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126240 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:32:45.135995 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126245 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:32:45.135995 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126249 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:32:45.135995 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126253 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:32:45.135995 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126257 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126261 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126265 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126270 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126274 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126278 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126282 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126287 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126292 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126296 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126300 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126305 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126309 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126313 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126318 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126322 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126326 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126330 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126334 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126338 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:32:45.136457 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126342 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:32:45.137033 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126346 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:32:45.137033 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126350 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:32:45.137033 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126357 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:32:45.137033 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126361 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:32:45.137033 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126365 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:32:45.137033 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126368 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:32:45.137033 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126376 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:32:45.137033 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126382 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:32:45.137033 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126387 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:32:45.137033 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126392 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:32:45.137033 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126396 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:32:45.137033 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126400 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:32:45.137033 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:45.126404 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:32:45.137033 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.126471 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 17:32:45.137033 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.129596 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 17:32:45.137033 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.130454 2579 server.go:1014] "Starting client certificate rotation" Nov 02 17:32:45.137033 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.130553 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:32:45.137033 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.130591 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:32:45.137460 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.132472 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:32:45.140746 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.140726 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:32:45.152737 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.152715 2579 log.go:25] "Validated CRI v1 runtime API" Nov 02 17:32:45.156968 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.156952 2579 log.go:25] "Validated CRI v1 image API" Nov 02 17:32:45.157827 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.157805 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 17:32:45.162687 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.162669 2579 fs.go:135] Filesystem UUIDs: map[5ff19fdd-8fb7-4c34-b8b9-284ff3d1a8ec:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 921a8f18-a4a4-4bde-9044-b365c11be3b1:/dev/nvme0n1p3] Nov 02 17:32:45.162747 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.162686 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 17:32:45.163536 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.163519 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 17:32:45.168358 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.168241 2579 manager.go:217] Machine: {Timestamp:2026-11-02 17:32:45.166515754 +0000 UTC m=+0.380089177 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092520 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22aa1c762d8ea629a72adce91f0c8b SystemUUID:ec22aa1c-762d-8ea6-29a7-2adce91f0c8b BootID:7a7ca67d-c634-4e85-a310-6a573ae23c48 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d5:5b:69:53 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d5:5b:69:53 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:54:15:96:ca:4e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 17:32:45.168358 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.168358 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 17:32:45.168465 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.168436 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 17:32:45.171970 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.171948 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 17:32:45.172109 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.171972 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-169.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 17:32:45.172653 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.172643 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 17:32:45.172683 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.172656 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 17:32:45.172683 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.172674 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:32:45.172740 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.172707 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:32:45.173502 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.173492 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 17:32:45.177447 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.177435 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 17:32:45.177571 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.177561 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 17:32:45.177604 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.177596 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 02 17:32:45.177650 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.177612 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 17:32:45.177650 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.177637 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 17:32:45.177704 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.177661 2579 kubelet.go:405] "Adding apiserver pod source" Nov 02 17:32:45.177704 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.177676 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 17:32:45.178165 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.178144 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:32:45.181426 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.181405 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 17:32:45.182437 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.182419 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cnxs6" Nov 02 17:32:45.186596 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.186580 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:32:45.188115 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.188093 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 17:32:45.188193 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.188127 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 17:32:45.188284 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.188272 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 17:32:45.188330 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.188290 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 17:32:45.188330 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.188297 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 17:32:45.188330 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.188303 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 17:32:45.188330 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.188310 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 17:32:45.188330 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.188316 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 17:32:45.189163 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.189144 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 17:32:45.189231 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.189175 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 17:32:45.189231 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.189189 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 17:32:45.189231 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.189203 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 17:32:45.189231 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.189220 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 17:32:45.189425 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.189237 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 17:32:45.189425 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.189263 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 17:32:45.189425 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.189274 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 17:32:45.190266 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.190244 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cnxs6" Nov 02 17:32:45.194176 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.194160 2579 server.go:1263] "Started kubelet" Nov 02 17:32:45.194275 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.194229 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 17:32:45.194317 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.194248 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 17:32:45.194317 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.194313 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 17:32:45.195004 ip-10-0-141-169 systemd[1]: Started Kubernetes Kubelet. Nov 02 17:32:45.198256 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.196395 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 02 17:32:45.198780 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.198766 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 17:32:45.200780 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.200764 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:32:45.202410 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:45.202394 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 17:32:45.202973 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.202960 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:32:45.204497 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.204479 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 17:32:45.204576 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.204497 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 17:32:45.205021 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:45.204640 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-169.us-east-2.compute.internal\" not found" Nov 02 17:32:45.205217 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.205204 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 17:32:45.205217 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.205211 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 17:32:45.205348 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.205221 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 17:32:45.205348 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.205339 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 02 17:32:45.205348 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.205347 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 02 17:32:45.205484 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.205413 2579 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-141-169.us-east-2.compute.internal" not found Nov 02 17:32:45.205843 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.205823 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 17:32:45.205843 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.205842 2579 factory.go:55] Registering systemd factory Nov 02 17:32:45.205973 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.205849 2579 factory.go:223] Registration of the systemd container factory successfully Nov 02 17:32:45.206337 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.206093 2579 factory.go:153] Registering CRI-O factory Nov 02 17:32:45.206337 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.206127 2579 factory.go:223] Registration of the crio container factory successfully Nov 02 17:32:45.206337 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.206146 2579 factory.go:103] Registering Raw factory Nov 02 17:32:45.206337 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.206161 2579 manager.go:1199] Started watching for new ooms in manager Nov 02 17:32:45.207396 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.207375 2579 manager.go:319] Starting recovery of all containers Nov 02 17:32:45.208082 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.208062 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:32:45.216847 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:45.216713 2579 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-169.us-east-2.compute.internal\" not found" node="ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:32:45.219059 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.219036 2579 manager.go:324] Recovery completed Nov 02 17:32:45.220134 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.220115 2579 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-141-169.us-east-2.compute.internal" not found Nov 02 17:32:45.224432 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.224417 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:32:45.226140 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.226124 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-169.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:32:45.226213 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.226151 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-169.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:32:45.226213 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.226165 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-169.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:32:45.226689 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.226673 2579 cpu_manager.go:226] "Starting" policy="none" Nov 02 17:32:45.226689 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.226682 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:32:45.226689 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.226685 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 17:32:45.226860 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.226701 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 17:32:45.227347 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.227333 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-169.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:32:45.227404 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.227372 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-169.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:32:45.227404 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.227389 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-169.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:32:45.228939 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.228926 2579 policy_none.go:50] "Start" Nov 02 17:32:45.228989 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.228941 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 17:32:45.228989 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.228954 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 17:32:45.231016 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.231004 2579 policy_none.go:44] "Start" Nov 02 17:32:45.264287 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.264163 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 17:32:45.268120 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.268108 2579 manager.go:341] "Starting Device Plugin manager" Nov 02 17:32:45.268576 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:45.268562 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 17:32:45.268576 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.268576 2579 server.go:87] "Starting device plugin registration server" Nov 02 17:32:45.268815 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.268803 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 17:32:45.268867 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.268817 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 17:32:45.268943 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.268925 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 17:32:45.269349 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.269060 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 17:32:45.269349 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.269078 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 17:32:45.269457 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:45.269442 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 17:32:45.269492 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:45.269479 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-169.us-east-2.compute.internal\" not found" Nov 02 17:32:45.274761 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.274745 2579 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-141-169.us-east-2.compute.internal" not found Nov 02 17:32:45.369218 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.369158 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:32:45.370191 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.370166 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-169.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:32:45.370266 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.370204 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-169.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:32:45.370266 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.370219 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-169.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:32:45.370266 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.370257 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:32:45.379956 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.379940 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:32:45.481784 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.481749 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 17:32:45.483700 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.482978 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 17:32:45.483700 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.483001 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 17:32:45.483700 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.483023 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 17:32:45.483700 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:45.483054 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 17:32:45.492354 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.492337 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:32:45.583926 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.583882 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal"] Nov 02 17:32:45.586176 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.586159 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:32:45.587861 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.587840 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:32:45.608100 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.608081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f82e5ebcf2f8d3b5e95e3af83d2365db-config\") pod \"kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal\" (UID: \"f82e5ebcf2f8d3b5e95e3af83d2365db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:32:45.608187 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.608107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca96163b2bc1e65e3d055163a6272914-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal\" (UID: \"ca96163b2bc1e65e3d055163a6272914\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:32:45.608187 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.608123 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca96163b2bc1e65e3d055163a6272914-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal\" (UID: \"ca96163b2bc1e65e3d055163a6272914\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:32:45.617963 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.617944 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:32:45.620690 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.620657 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:32:45.632304 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.632286 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:32:45.632363 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.632303 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:32:45.708795 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.708770 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca96163b2bc1e65e3d055163a6272914-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal\" (UID: \"ca96163b2bc1e65e3d055163a6272914\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:32:45.708795 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.708796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca96163b2bc1e65e3d055163a6272914-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal\" (UID: \"ca96163b2bc1e65e3d055163a6272914\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:32:45.708975 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.708812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f82e5ebcf2f8d3b5e95e3af83d2365db-config\") pod \"kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal\" (UID: \"f82e5ebcf2f8d3b5e95e3af83d2365db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:32:45.708975 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.708839 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f82e5ebcf2f8d3b5e95e3af83d2365db-config\") pod \"kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal\" (UID: \"f82e5ebcf2f8d3b5e95e3af83d2365db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:32:45.708975 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.708856 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca96163b2bc1e65e3d055163a6272914-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal\" (UID: \"ca96163b2bc1e65e3d055163a6272914\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:32:45.708975 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.708892 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca96163b2bc1e65e3d055163a6272914-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal\" (UID: \"ca96163b2bc1e65e3d055163a6272914\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:32:45.937305 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.937210 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:32:45.937305 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:45.937241 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:32:46.130995 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.130952 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 17:32:46.131607 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.131128 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 17:32:46.131607 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.131159 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 17:32:46.131607 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.131198 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 17:32:46.180663 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.180634 2579 apiserver.go:52] "Watching apiserver" Nov 02 17:32:46.196478 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.196399 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:27:46 +0000 UTC" deadline="2028-07-20 09:08:33.872346816 +0000 UTC" Nov 02 17:32:46.196478 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.196431 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15015h35m47.675918828s" Nov 02 17:32:46.197305 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.197290 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 17:32:46.197593 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.197576 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-vdll2","openshift-network-diagnostics/network-check-target-424d5","kube-system/global-pull-secret-syncer-f4xwb","openshift-dns/node-resolver-5tkc8","openshift-image-registry/node-ca-q28kf","openshift-ovn-kubernetes/ovnkube-node-5wh7f","kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal","openshift-multus/multus-gbc4r","openshift-multus/network-metrics-daemon-dfthz","kube-system/konnectivity-agent-cjnbh","openshift-network-operator/iptables-alerter-5l7cq","openshift-cluster-node-tuning-operator/tuned-s9kl9"] Nov 02 17:32:46.200258 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.200242 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.202456 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.202437 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:32:46.202550 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:46.202506 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:32:46.204820 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.204792 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 17:32:46.204937 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.204912 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:32:46.205036 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:46.205011 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:32:46.205317 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.205295 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 17:32:46.205387 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.205300 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 17:32:46.205387 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.205301 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 17:32:46.205824 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.205805 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 17:32:46.205893 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.205874 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 17:32:46.208616 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.208591 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5tkc8" Nov 02 17:32:46.209945 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.209922 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wjf7q\"" Nov 02 17:32:46.210917 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.210899 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q28kf" Nov 02 17:32:46.213198 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.213180 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d225fb4-f477-4971-bce1-8aaa24a79be8-cni-binary-copy\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.213198 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.213182 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.213360 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.213248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d225fb4-f477-4971-bce1-8aaa24a79be8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.213360 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.213288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9tmr\" (UniqueName: \"kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr\") pod \"network-check-target-424d5\" (UID: \"9ce34815-aef2-487a-afd1-fe81200a423d\") " pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:32:46.213360 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.213312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc-tmp-dir\") pod \"node-resolver-5tkc8\" (UID: \"a48715bf-e5aa-4b50-a9a5-4916e6fbefbc\") " pod="openshift-dns/node-resolver-5tkc8" Nov 02 17:32:46.213360 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.213343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d225fb4-f477-4971-bce1-8aaa24a79be8-system-cni-dir\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.213585 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.213376 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-dbus\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:32:46.213585 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.213404 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:32:46.213585 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.213433 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d225fb4-f477-4971-bce1-8aaa24a79be8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.213585 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.213499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d225fb4-f477-4971-bce1-8aaa24a79be8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.213585 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.213535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-kubelet-config\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:32:46.213585 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.213583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc-hosts-file\") pod \"node-resolver-5tkc8\" (UID: \"a48715bf-e5aa-4b50-a9a5-4916e6fbefbc\") " pod="openshift-dns/node-resolver-5tkc8" Nov 02 17:32:46.213856 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.213621 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4662z\" (UniqueName: \"kubernetes.io/projected/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc-kube-api-access-4662z\") pod \"node-resolver-5tkc8\" (UID: \"a48715bf-e5aa-4b50-a9a5-4916e6fbefbc\") " pod="openshift-dns/node-resolver-5tkc8" Nov 02 17:32:46.213856 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.213676 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d225fb4-f477-4971-bce1-8aaa24a79be8-cnibin\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.213856 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.213720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwnst\" (UniqueName: \"kubernetes.io/projected/8d225fb4-f477-4971-bce1-8aaa24a79be8-kube-api-access-kwnst\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.215562 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.215544 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.217945 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.217924 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.220451 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.220425 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:32:46.220556 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:46.220509 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:32:46.222779 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.222763 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cjnbh" Nov 02 17:32:46.225202 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.225184 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5l7cq" Nov 02 17:32:46.228183 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.228052 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.246857 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.246833 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 17:32:46.246940 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.246927 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-k8lqb" Nov 02 17:32:46.256664 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.256643 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k8lqb" Nov 02 17:32:46.307077 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.307056 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 17:32:46.313801 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.313782 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 17:32:46.313943 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.313801 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 17:32:46.313943 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.313855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tmr\" (UniqueName: \"kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr\") pod \"network-check-target-424d5\" (UID: \"9ce34815-aef2-487a-afd1-fe81200a423d\") " pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:32:46.313943 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.313880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-log-socket\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.313943 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.313897 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnt46\" (UniqueName: \"kubernetes.io/projected/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-kube-api-access-cnt46\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.313943 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.313934 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 17:32:46.314215 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.313963 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:32:46.314215 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.313967 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 17:32:46.314215 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.313972 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c00d16e7-b92c-46c7-a68f-797274bbef93-konnectivity-ca\") pod \"konnectivity-agent-cjnbh\" (UID: \"c00d16e7-b92c-46c7-a68f-797274bbef93\") " pod="kube-system/konnectivity-agent-cjnbh" Nov 02 17:32:46.314215 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314121 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 17:32:46.314215 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314136 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/44174ea2-6704-423f-a176-c3abe64be85e-iptables-alerter-script\") pod \"iptables-alerter-5l7cq\" (UID: \"44174ea2-6704-423f-a176-c3abe64be85e\") " pod="openshift-network-operator/iptables-alerter-5l7cq" Nov 02 17:32:46.314215 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314151 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xjkc9\"" Nov 02 17:32:46.314215 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314198 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-kubernetes\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.314548 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314252 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-sys\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.314548 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314274 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-host\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.314548 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:32:46.314548 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-var-lib-openvswitch\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.314548 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314353 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 17:32:46.314548 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314382 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-tuned\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.314548 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314389 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 17:32:46.314548 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314433 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:32:46.314548 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314435 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-tmp\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.314548 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314492 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 17:32:46.314548 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:46.314509 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:46.314548 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314501 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k78hz\"" Nov 02 17:32:46.315049 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314562 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-run-multus-certs\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.315049 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314638 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 17:32:46.315049 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:46.314665 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret podName:d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:46.814612464 +0000 UTC m=+2.028185906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret") pod "global-pull-secret-syncer-f4xwb" (UID: "d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:46.315049 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d225fb4-f477-4971-bce1-8aaa24a79be8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.315049 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314721 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-kubelet-config\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:32:46.315049 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4662z\" (UniqueName: \"kubernetes.io/projected/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc-kube-api-access-4662z\") pod \"node-resolver-5tkc8\" (UID: \"a48715bf-e5aa-4b50-a9a5-4916e6fbefbc\") " pod="openshift-dns/node-resolver-5tkc8" Nov 02 17:32:46.315049 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-run-systemd\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.315049 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314797 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-etc-openvswitch\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.315049 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314822 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-system-cni-dir\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.315049 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-run-ovn\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.315049 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314873 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-var-lib-kubelet\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.315049 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314931 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t97rb\" (UniqueName: \"kubernetes.io/projected/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-kube-api-access-t97rb\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.315049 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314974 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-var-lib-cni-bin\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.315049 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.314972 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d225fb4-f477-4971-bce1-8aaa24a79be8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.315049 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315018 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-kubelet-config\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:32:46.315644 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315065 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-conf-dir\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.315644 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315113 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-systemd-units\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.315644 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315173 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-cni-bin\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.315644 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315194 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-cni-netd\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.315644 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315239 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-device-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.315644 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315273 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-etc-selinux\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.315644 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-var-lib-kubelet\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.315644 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315340 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d225fb4-f477-4971-bce1-8aaa24a79be8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.315644 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315367 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-run-openvswitch\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.315644 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315384 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c00d16e7-b92c-46c7-a68f-797274bbef93-agent-certs\") pod \"konnectivity-agent-cjnbh\" (UID: \"c00d16e7-b92c-46c7-a68f-797274bbef93\") " pod="kube-system/konnectivity-agent-cjnbh" Nov 02 17:32:46.315644 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315400 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-run\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.315644 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d225fb4-f477-4971-bce1-8aaa24a79be8-cnibin\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.315644 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315459 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-kubelet\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.315644 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315484 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d225fb4-f477-4971-bce1-8aaa24a79be8-cnibin\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.315644 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-slash\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.315644 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315594 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 17:32:46.315644 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315606 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-lib-modules\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315664 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315691 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-cni-dir\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315728 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-run-k8s-cni-cncf-io\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315758 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-var-lib-cni-multus\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315801 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlzk8\" (UniqueName: \"kubernetes.io/projected/3e35ffc8-ef10-4532-af91-38841c3f4581-kube-api-access-mlzk8\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315822 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315828 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d225fb4-f477-4971-bce1-8aaa24a79be8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315858 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315860 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315919 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc-tmp-dir\") pod \"node-resolver-5tkc8\" (UID: \"a48715bf-e5aa-4b50-a9a5-4916e6fbefbc\") " pod="openshift-dns/node-resolver-5tkc8" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315928 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d225fb4-f477-4971-bce1-8aaa24a79be8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.315948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-env-overrides\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316024 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-sysctl-d\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316035 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316058 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-cnibin\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316067 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316124 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-hostroot\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316149 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-daemon-config\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316165 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-etc-kubernetes\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316174 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dtld9\"" Nov 02 17:32:46.316328 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316193 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-dbus\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:32:46.317187 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316213 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-node-log\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.317187 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316227 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc-tmp-dir\") pod \"node-resolver-5tkc8\" (UID: \"a48715bf-e5aa-4b50-a9a5-4916e6fbefbc\") " pod="openshift-dns/node-resolver-5tkc8" Nov 02 17:32:46.317187 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316274 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-ovnkube-config\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.317187 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95kp2\" (UniqueName: \"kubernetes.io/projected/44174ea2-6704-423f-a176-c3abe64be85e-kube-api-access-95kp2\") pod \"iptables-alerter-5l7cq\" (UID: \"44174ea2-6704-423f-a176-c3abe64be85e\") " pod="openshift-network-operator/iptables-alerter-5l7cq" Nov 02 17:32:46.317187 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316304 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d225fb4-f477-4971-bce1-8aaa24a79be8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.317187 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316358 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-modprobe-d\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.317187 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316378 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.317187 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316396 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-socket-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.317187 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316412 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e35ffc8-ef10-4532-af91-38841c3f4581-cni-binary-copy\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.317187 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316423 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-dbus\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:32:46.317187 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwnst\" (UniqueName: \"kubernetes.io/projected/8d225fb4-f477-4971-bce1-8aaa24a79be8-kube-api-access-kwnst\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.317187 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316460 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-sysctl-conf\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.317187 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316547 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4lln\" (UniqueName: \"kubernetes.io/projected/d9300fb6-6926-4429-8797-603dfb361818-kube-api-access-m4lln\") pod \"node-ca-q28kf\" (UID: \"d9300fb6-6926-4429-8797-603dfb361818\") " pod="openshift-image-registry/node-ca-q28kf" Nov 02 17:32:46.317187 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316605 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs\") pod \"network-metrics-daemon-dfthz\" (UID: \"86286bdb-d69d-481a-b874-4ff3df42906f\") " pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:32:46.317187 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqxvs\" (UniqueName: \"kubernetes.io/projected/86286bdb-d69d-481a-b874-4ff3df42906f-kube-api-access-lqxvs\") pod \"network-metrics-daemon-dfthz\" (UID: \"86286bdb-d69d-481a-b874-4ff3df42906f\") " pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:32:46.317187 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316678 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d225fb4-f477-4971-bce1-8aaa24a79be8-cni-binary-copy\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.317795 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-ovn-node-metrics-cert\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.317795 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9300fb6-6926-4429-8797-603dfb361818-host\") pod \"node-ca-q28kf\" (UID: \"d9300fb6-6926-4429-8797-603dfb361818\") " pod="openshift-image-registry/node-ca-q28kf" Nov 02 17:32:46.317795 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d9300fb6-6926-4429-8797-603dfb361818-serviceca\") pod \"node-ca-q28kf\" (UID: \"d9300fb6-6926-4429-8797-603dfb361818\") " pod="openshift-image-registry/node-ca-q28kf" Nov 02 17:32:46.317795 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-socket-dir-parent\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.317795 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d225fb4-f477-4971-bce1-8aaa24a79be8-system-cni-dir\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.317795 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316902 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-run-netns\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.317795 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316908 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d225fb4-f477-4971-bce1-8aaa24a79be8-system-cni-dir\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.317795 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316937 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.317795 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.316960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.317795 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.317040 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc-hosts-file\") pod \"node-resolver-5tkc8\" (UID: \"a48715bf-e5aa-4b50-a9a5-4916e6fbefbc\") " pod="openshift-dns/node-resolver-5tkc8" Nov 02 17:32:46.317795 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.317073 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-ovnkube-script-lib\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.317795 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.317082 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d225fb4-f477-4971-bce1-8aaa24a79be8-cni-binary-copy\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.317795 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.317097 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc-hosts-file\") pod \"node-resolver-5tkc8\" (UID: \"a48715bf-e5aa-4b50-a9a5-4916e6fbefbc\") " pod="openshift-dns/node-resolver-5tkc8" Nov 02 17:32:46.317795 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.317100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-sysconfig\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.317795 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.317153 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-systemd\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.317795 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.317184 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-registration-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.318472 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.317202 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-sys-fs\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.318472 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.317219 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44174ea2-6704-423f-a176-c3abe64be85e-host-slash\") pod \"iptables-alerter-5l7cq\" (UID: \"44174ea2-6704-423f-a176-c3abe64be85e\") " pod="openshift-network-operator/iptables-alerter-5l7cq" Nov 02 17:32:46.318472 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.317257 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptckf\" (UniqueName: \"kubernetes.io/projected/40e6a790-de24-49f1-b507-ed74d1446868-kube-api-access-ptckf\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.318472 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.317286 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-run-netns\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.318472 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.317529 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 17:32:46.318472 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.317530 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lkfrw\"" Nov 02 17:32:46.318472 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.317739 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ttfxd\"" Nov 02 17:32:46.318472 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.317837 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sbqnj\"" Nov 02 17:32:46.318472 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.317845 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 17:32:46.318472 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.317855 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 17:32:46.318472 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.317862 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jl4vs\"" Nov 02 17:32:46.318472 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.317865 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zg2g5\"" Nov 02 17:32:46.318472 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.318014 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 17:32:46.320967 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:46.320952 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:32:46.320967 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:46.320969 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:32:46.321078 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:46.320980 2579 projected.go:196] Error preparing data for projected volume kube-api-access-m9tmr for pod openshift-network-diagnostics/network-check-target-424d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:46.321078 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:46.321026 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr podName:9ce34815-aef2-487a-afd1-fe81200a423d nodeName:}" failed. No retries permitted until 2026-11-02 17:32:46.821013959 +0000 UTC m=+2.034587401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m9tmr" (UniqueName: "kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr") pod "network-check-target-424d5" (UID: "9ce34815-aef2-487a-afd1-fe81200a423d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:46.321506 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.321476 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 17:32:46.324847 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.324826 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwnst\" (UniqueName: \"kubernetes.io/projected/8d225fb4-f477-4971-bce1-8aaa24a79be8-kube-api-access-kwnst\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.324932 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.324855 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4662z\" (UniqueName: \"kubernetes.io/projected/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc-kube-api-access-4662z\") pod \"node-resolver-5tkc8\" (UID: \"a48715bf-e5aa-4b50-a9a5-4916e6fbefbc\") " pod="openshift-dns/node-resolver-5tkc8" Nov 02 17:32:46.417896 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.417856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-kubernetes\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.418070 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.417909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-sys\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.418070 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.417967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-host\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.418070 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-var-lib-openvswitch\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.418070 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-tuned\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.418293 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-tmp\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.418293 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-run-multus-certs\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.418293 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418141 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-run-systemd\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.418293 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418282 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-etc-openvswitch\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.418479 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418308 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-system-cni-dir\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.418479 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418338 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-run-ovn\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.418479 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-var-lib-kubelet\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.418479 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t97rb\" (UniqueName: \"kubernetes.io/projected/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-kube-api-access-t97rb\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.418479 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-var-lib-cni-bin\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.418748 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418474 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-conf-dir\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.418748 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418525 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-systemd-units\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.418748 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-cni-bin\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.418748 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-cni-netd\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.418748 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418721 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-device-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.418748 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-etc-selinux\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.419040 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-var-lib-kubelet\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.419040 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-run-openvswitch\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.419040 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418893 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c00d16e7-b92c-46c7-a68f-797274bbef93-agent-certs\") pod \"konnectivity-agent-cjnbh\" (UID: \"c00d16e7-b92c-46c7-a68f-797274bbef93\") " pod="kube-system/konnectivity-agent-cjnbh" Nov 02 17:32:46.419040 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.418937 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-run\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.419245 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.419075 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-kubelet\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.419245 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.419132 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-slash\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.419245 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.419179 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-lib-modules\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.419245 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.419208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-cni-dir\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.419507 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.419252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-run-k8s-cni-cncf-io\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.419507 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.419277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-var-lib-cni-multus\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.419507 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.419307 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlzk8\" (UniqueName: \"kubernetes.io/projected/3e35ffc8-ef10-4532-af91-38841c3f4581-kube-api-access-mlzk8\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.419507 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.419366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-env-overrides\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.419507 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.419440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-sysctl-d\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.419507 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.419482 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-cnibin\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.419833 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.419512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-hostroot\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.419833 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.419576 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-daemon-config\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.419833 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.419624 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-etc-kubernetes\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.419833 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.419657 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-node-log\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.419833 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.419686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-ovnkube-config\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.419833 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.419740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95kp2\" (UniqueName: \"kubernetes.io/projected/44174ea2-6704-423f-a176-c3abe64be85e-kube-api-access-95kp2\") pod \"iptables-alerter-5l7cq\" (UID: \"44174ea2-6704-423f-a176-c3abe64be85e\") " pod="openshift-network-operator/iptables-alerter-5l7cq" Nov 02 17:32:46.420134 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.419890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-modprobe-d\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.420134 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.419923 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.420134 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.419954 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-socket-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.420134 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420010 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e35ffc8-ef10-4532-af91-38841c3f4581-cni-binary-copy\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.420134 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420043 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-sysctl-conf\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.420134 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4lln\" (UniqueName: \"kubernetes.io/projected/d9300fb6-6926-4429-8797-603dfb361818-kube-api-access-m4lln\") pod \"node-ca-q28kf\" (UID: \"d9300fb6-6926-4429-8797-603dfb361818\") " pod="openshift-image-registry/node-ca-q28kf" Nov 02 17:32:46.420134 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs\") pod \"network-metrics-daemon-dfthz\" (UID: \"86286bdb-d69d-481a-b874-4ff3df42906f\") " pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:32:46.420473 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420147 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqxvs\" (UniqueName: \"kubernetes.io/projected/86286bdb-d69d-481a-b874-4ff3df42906f-kube-api-access-lqxvs\") pod \"network-metrics-daemon-dfthz\" (UID: \"86286bdb-d69d-481a-b874-4ff3df42906f\") " pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:32:46.420473 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420193 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-ovn-node-metrics-cert\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.422769 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420326 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-var-lib-openvswitch\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.422769 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420479 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-run-openvswitch\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.422769 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.421375 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-sysctl-conf\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.422769 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420579 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-cni-bin\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.422769 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420648 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-etc-openvswitch\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.422769 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420686 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-run-systemd\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.422769 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.421467 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-modprobe-d\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.422769 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420697 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-var-lib-kubelet\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.422769 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420641 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-cni-netd\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.422769 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420815 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-system-cni-dir\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.422769 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.421538 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-host\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.422769 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420847 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-var-lib-cni-bin\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.422769 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420869 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-var-lib-kubelet\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.422769 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420873 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-conf-dir\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.422769 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420902 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-device-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.422769 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420913 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-node-log\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.422769 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420954 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-run-k8s-cni-cncf-io\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.422769 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.421013 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-var-lib-cni-multus\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.423760 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.421028 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-socket-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.423760 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.421090 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.423760 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.421213 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9300fb6-6926-4429-8797-603dfb361818-host\") pod \"node-ca-q28kf\" (UID: \"d9300fb6-6926-4429-8797-603dfb361818\") " pod="openshift-image-registry/node-ca-q28kf" Nov 02 17:32:46.423760 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.421259 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-run-multus-certs\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.423760 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.421334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9300fb6-6926-4429-8797-603dfb361818-host\") pod \"node-ca-q28kf\" (UID: \"d9300fb6-6926-4429-8797-603dfb361818\") " pod="openshift-image-registry/node-ca-q28kf" Nov 02 17:32:46.423760 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.421345 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-lib-modules\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.423760 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.421355 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-cni-dir\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.423760 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420784 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-systemd-units\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.423760 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420808 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-etc-selinux\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.423760 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.420827 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-run-ovn\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.423760 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.421525 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-kubernetes\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.423760 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:46.421711 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:46.423760 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.421774 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-tuned\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.423760 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.421861 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-ovnkube-config\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.423760 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.421925 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e35ffc8-ef10-4532-af91-38841c3f4581-cni-binary-copy\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.423760 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.421926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d9300fb6-6926-4429-8797-603dfb361818-serviceca\") pod \"node-ca-q28kf\" (UID: \"d9300fb6-6926-4429-8797-603dfb361818\") " pod="openshift-image-registry/node-ca-q28kf" Nov 02 17:32:46.423760 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.421967 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-cnibin\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.423760 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.421989 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-kubelet\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.423760 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422001 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-socket-dir-parent\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.424655 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:46.422081 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs podName:86286bdb-d69d-481a-b874-4ff3df42906f nodeName:}" failed. No retries permitted until 2026-11-02 17:32:46.922058347 +0000 UTC m=+2.135631759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs") pod "network-metrics-daemon-dfthz" (UID: "86286bdb-d69d-481a-b874-4ff3df42906f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:46.424655 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422080 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-etc-kubernetes\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.424655 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422107 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-run\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.424655 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422158 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-run-netns\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.424655 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422164 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-sysctl-d\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.424655 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422243 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-sys\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.424655 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422307 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-hostroot\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.424655 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422450 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.424655 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422480 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-socket-dir-parent\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.424655 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422525 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.424655 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422556 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-slash\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.424655 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422613 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.424655 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-ovnkube-script-lib\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.424655 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422668 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d9300fb6-6926-4429-8797-603dfb361818-serviceca\") pod \"node-ca-q28kf\" (UID: \"d9300fb6-6926-4429-8797-603dfb361818\") " pod="openshift-image-registry/node-ca-q28kf" Nov 02 17:32:46.424655 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422670 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.424655 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422726 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-env-overrides\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.424655 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-sysconfig\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.425303 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422735 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-run-netns\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.425303 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422767 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-daemon-config\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.425303 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-systemd\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.425303 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422930 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-systemd\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.425303 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.422933 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-registration-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.425303 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.423001 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-sysconfig\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.425303 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.423006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-sys-fs\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.425303 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.423036 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-registration-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.425303 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.423191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44174ea2-6704-423f-a176-c3abe64be85e-host-slash\") pod \"iptables-alerter-5l7cq\" (UID: \"44174ea2-6704-423f-a176-c3abe64be85e\") " pod="openshift-network-operator/iptables-alerter-5l7cq" Nov 02 17:32:46.425303 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.423345 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptckf\" (UniqueName: \"kubernetes.io/projected/40e6a790-de24-49f1-b507-ed74d1446868-kube-api-access-ptckf\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.425303 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.423865 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44174ea2-6704-423f-a176-c3abe64be85e-host-slash\") pod \"iptables-alerter-5l7cq\" (UID: \"44174ea2-6704-423f-a176-c3abe64be85e\") " pod="openshift-network-operator/iptables-alerter-5l7cq" Nov 02 17:32:46.425303 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.423883 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-sys-fs\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.425303 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.423936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-run-netns\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.425303 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.423993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-log-socket\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.425303 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.424024 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-ovnkube-script-lib\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.425303 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.424043 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-ovn-node-metrics-cert\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.425303 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.424046 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnt46\" (UniqueName: \"kubernetes.io/projected/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-kube-api-access-cnt46\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.425817 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.424089 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-log-socket\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.425817 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.424095 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-run-netns\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.425817 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.424147 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c00d16e7-b92c-46c7-a68f-797274bbef93-konnectivity-ca\") pod \"konnectivity-agent-cjnbh\" (UID: \"c00d16e7-b92c-46c7-a68f-797274bbef93\") " pod="kube-system/konnectivity-agent-cjnbh" Nov 02 17:32:46.425817 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.424245 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/44174ea2-6704-423f-a176-c3abe64be85e-iptables-alerter-script\") pod \"iptables-alerter-5l7cq\" (UID: \"44174ea2-6704-423f-a176-c3abe64be85e\") " pod="openshift-network-operator/iptables-alerter-5l7cq" Nov 02 17:32:46.425817 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.424726 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c00d16e7-b92c-46c7-a68f-797274bbef93-agent-certs\") pod \"konnectivity-agent-cjnbh\" (UID: \"c00d16e7-b92c-46c7-a68f-797274bbef93\") " pod="kube-system/konnectivity-agent-cjnbh" Nov 02 17:32:46.425817 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.424735 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c00d16e7-b92c-46c7-a68f-797274bbef93-konnectivity-ca\") pod \"konnectivity-agent-cjnbh\" (UID: \"c00d16e7-b92c-46c7-a68f-797274bbef93\") " pod="kube-system/konnectivity-agent-cjnbh" Nov 02 17:32:46.425817 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.424820 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/44174ea2-6704-423f-a176-c3abe64be85e-iptables-alerter-script\") pod \"iptables-alerter-5l7cq\" (UID: \"44174ea2-6704-423f-a176-c3abe64be85e\") " pod="openshift-network-operator/iptables-alerter-5l7cq" Nov 02 17:32:46.425817 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.425194 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-tmp\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.434155 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.434081 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlzk8\" (UniqueName: \"kubernetes.io/projected/3e35ffc8-ef10-4532-af91-38841c3f4581-kube-api-access-mlzk8\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.435045 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.434550 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4lln\" (UniqueName: \"kubernetes.io/projected/d9300fb6-6926-4429-8797-603dfb361818-kube-api-access-m4lln\") pod \"node-ca-q28kf\" (UID: \"d9300fb6-6926-4429-8797-603dfb361818\") " pod="openshift-image-registry/node-ca-q28kf" Nov 02 17:32:46.435045 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.434970 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95kp2\" (UniqueName: \"kubernetes.io/projected/44174ea2-6704-423f-a176-c3abe64be85e-kube-api-access-95kp2\") pod \"iptables-alerter-5l7cq\" (UID: \"44174ea2-6704-423f-a176-c3abe64be85e\") " pod="openshift-network-operator/iptables-alerter-5l7cq" Nov 02 17:32:46.435221 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.435080 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t97rb\" (UniqueName: \"kubernetes.io/projected/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-kube-api-access-t97rb\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.435301 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.435280 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnt46\" (UniqueName: \"kubernetes.io/projected/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-kube-api-access-cnt46\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.435420 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.435398 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqxvs\" (UniqueName: \"kubernetes.io/projected/86286bdb-d69d-481a-b874-4ff3df42906f-kube-api-access-lqxvs\") pod \"network-metrics-daemon-dfthz\" (UID: \"86286bdb-d69d-481a-b874-4ff3df42906f\") " pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:32:46.436893 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.436871 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptckf\" (UniqueName: \"kubernetes.io/projected/40e6a790-de24-49f1-b507-ed74d1446868-kube-api-access-ptckf\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.542001 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.541933 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:32:46.550295 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.550247 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5tkc8" Nov 02 17:32:46.571169 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.571144 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q28kf" Nov 02 17:32:46.604064 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.604035 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:32:46.607838 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.607814 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:32:46.609992 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.609975 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gbc4r" Nov 02 17:32:46.629102 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.629075 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cjnbh" Nov 02 17:32:46.645640 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.645606 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5l7cq" Nov 02 17:32:46.647197 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:46.647163 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11bb6b24_b4e9_4764_a40a_e1daa2c613c2.slice/crio-ef1fd04818d9c3135fea78c802e9766f59d1009c48cf7395e53284f02f7ddded WatchSource:0}: Error finding container ef1fd04818d9c3135fea78c802e9766f59d1009c48cf7395e53284f02f7ddded: Status 404 returned error can't find the container with id ef1fd04818d9c3135fea78c802e9766f59d1009c48cf7395e53284f02f7ddded Nov 02 17:32:46.648355 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:46.648319 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e35ffc8_ef10_4532_af91_38841c3f4581.slice/crio-dcc69eae5c7774f8b56c3f671e21766dce7bf983a0337b36318d9b06cc76d13d WatchSource:0}: Error finding container dcc69eae5c7774f8b56c3f671e21766dce7bf983a0337b36318d9b06cc76d13d: Status 404 returned error can't find the container with id dcc69eae5c7774f8b56c3f671e21766dce7bf983a0337b36318d9b06cc76d13d Nov 02 17:32:46.648813 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:46.648785 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00d16e7_b92c_46c7_a68f_797274bbef93.slice/crio-2de52cc070fc04494fe6bae9e958fde642017d687e5bc27b5230393cefc5b6ca WatchSource:0}: Error finding container 2de52cc070fc04494fe6bae9e958fde642017d687e5bc27b5230393cefc5b6ca: Status 404 returned error can't find the container with id 2de52cc070fc04494fe6bae9e958fde642017d687e5bc27b5230393cefc5b6ca Nov 02 17:32:46.650452 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.650428 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:32:46.653970 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.653860 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:32:46.658421 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:46.658398 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44174ea2_6704_423f_a176_c3abe64be85e.slice/crio-7c2265955010341f47ae43e2242c473921e2d11089d45d940bfbc87218662811 WatchSource:0}: Error finding container 7c2265955010341f47ae43e2242c473921e2d11089d45d940bfbc87218662811: Status 404 returned error can't find the container with id 7c2265955010341f47ae43e2242c473921e2d11089d45d940bfbc87218662811 Nov 02 17:32:46.662382 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:46.662362 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f379fbd_7dcf_4bff_a4c0_d946e711cf7e.slice/crio-60721f952051cb1bc046bc98ed1ec34d6b209b1ff8f92ee98f36c6f6fd6f1cd1 WatchSource:0}: Error finding container 60721f952051cb1bc046bc98ed1ec34d6b209b1ff8f92ee98f36c6f6fd6f1cd1: Status 404 returned error can't find the container with id 60721f952051cb1bc046bc98ed1ec34d6b209b1ff8f92ee98f36c6f6fd6f1cd1 Nov 02 17:32:46.826806 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.826713 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tmr\" (UniqueName: \"kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr\") pod \"network-check-target-424d5\" (UID: \"9ce34815-aef2-487a-afd1-fe81200a423d\") " pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:32:46.826806 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.826756 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:32:46.826997 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:46.826865 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:46.826997 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:46.826876 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:32:46.826997 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:46.826901 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:32:46.826997 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:46.826912 2579 projected.go:196] Error preparing data for projected volume kube-api-access-m9tmr for pod openshift-network-diagnostics/network-check-target-424d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:46.826997 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:46.826922 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret podName:d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:47.826906722 +0000 UTC m=+3.040480149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret") pod "global-pull-secret-syncer-f4xwb" (UID: "d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:46.826997 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:46.826953 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr podName:9ce34815-aef2-487a-afd1-fe81200a423d nodeName:}" failed. No retries permitted until 2026-11-02 17:32:47.826940924 +0000 UTC m=+3.040514335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9tmr" (UniqueName: "kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr") pod "network-check-target-424d5" (UID: "9ce34815-aef2-487a-afd1-fe81200a423d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:46.928152 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.928115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs\") pod \"network-metrics-daemon-dfthz\" (UID: \"86286bdb-d69d-481a-b874-4ff3df42906f\") " pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:32:46.928347 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:46.928230 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:46.928347 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:46.928310 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs podName:86286bdb-d69d-481a-b874-4ff3df42906f nodeName:}" failed. No retries permitted until 2026-11-02 17:32:47.928289093 +0000 UTC m=+3.141862520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs") pod "network-metrics-daemon-dfthz" (UID: "86286bdb-d69d-481a-b874-4ff3df42906f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:46.955037 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:46.955011 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:32:46.961549 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:46.961521 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48715bf_e5aa_4b50_a9a5_4916e6fbefbc.slice/crio-d8567d7af3f898368be2a2d0cbb7633ab84ea10911fa1f6c561d05bb3d8d371b WatchSource:0}: Error finding container d8567d7af3f898368be2a2d0cbb7633ab84ea10911fa1f6c561d05bb3d8d371b: Status 404 returned error can't find the container with id d8567d7af3f898368be2a2d0cbb7633ab84ea10911fa1f6c561d05bb3d8d371b Nov 02 17:32:47.038245 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:32:47.038188 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d225fb4_f477_4971_bce1_8aaa24a79be8.slice/crio-ec0c7d25ba197d2d809535834a87240820ff73dde8bd8883fd36a6279c5cd644 WatchSource:0}: Error finding container ec0c7d25ba197d2d809535834a87240820ff73dde8bd8883fd36a6279c5cd644: Status 404 returned error can't find the container with id ec0c7d25ba197d2d809535834a87240820ff73dde8bd8883fd36a6279c5cd644 Nov 02 17:32:47.260374 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:47.260198 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:27:47 +0000 UTC" deadline="2028-07-15 06:26:20.830428226 +0000 UTC" Nov 02 17:32:47.260374 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:47.260322 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14892h53m33.570111449s" Nov 02 17:32:47.475303 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:47.475269 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:32:47.504525 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:47.504464 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdll2" event={"ID":"8d225fb4-f477-4971-bce1-8aaa24a79be8","Type":"ContainerStarted","Data":"ec0c7d25ba197d2d809535834a87240820ff73dde8bd8883fd36a6279c5cd644"} Nov 02 17:32:47.524771 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:47.524676 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q28kf" event={"ID":"d9300fb6-6926-4429-8797-603dfb361818","Type":"ContainerStarted","Data":"bac27cb3c407150e65218780c36da0ea0a465950911c6e1ef696403ab454bc4d"} Nov 02 17:32:47.537475 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:47.537439 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" event={"ID":"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e","Type":"ContainerStarted","Data":"60721f952051cb1bc046bc98ed1ec34d6b209b1ff8f92ee98f36c6f6fd6f1cd1"} Nov 02 17:32:47.543075 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:47.543036 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:32:47.551648 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:47.551611 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5l7cq" event={"ID":"44174ea2-6704-423f-a176-c3abe64be85e","Type":"ContainerStarted","Data":"7c2265955010341f47ae43e2242c473921e2d11089d45d940bfbc87218662811"} Nov 02 17:32:47.556956 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:47.556925 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cjnbh" event={"ID":"c00d16e7-b92c-46c7-a68f-797274bbef93","Type":"ContainerStarted","Data":"2de52cc070fc04494fe6bae9e958fde642017d687e5bc27b5230393cefc5b6ca"} Nov 02 17:32:47.569954 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:47.569548 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" event={"ID":"ca96163b2bc1e65e3d055163a6272914","Type":"ContainerStarted","Data":"7c97a3b904cb02796a1e696cc3a9a28c0baa6b091b3366784a6b7c8c039a7488"} Nov 02 17:32:47.606650 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:47.606608 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5tkc8" event={"ID":"a48715bf-e5aa-4b50-a9a5-4916e6fbefbc","Type":"ContainerStarted","Data":"d8567d7af3f898368be2a2d0cbb7633ab84ea10911fa1f6c561d05bb3d8d371b"} Nov 02 17:32:47.609527 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:47.609455 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" event={"ID":"11bb6b24-b4e9-4764-a40a-e1daa2c613c2","Type":"ContainerStarted","Data":"ef1fd04818d9c3135fea78c802e9766f59d1009c48cf7395e53284f02f7ddded"} Nov 02 17:32:47.622931 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:47.622894 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal" event={"ID":"f82e5ebcf2f8d3b5e95e3af83d2365db","Type":"ContainerStarted","Data":"c1cacef195d5be9ba06c4120bc166fa099b012a5b5e9eadce24a287821c1064f"} Nov 02 17:32:47.638719 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:47.638613 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" event={"ID":"40e6a790-de24-49f1-b507-ed74d1446868","Type":"ContainerStarted","Data":"06a75e22443636b5d6f9595c1a91faf891a1814859883ba9d367b81fab5ae7f3"} Nov 02 17:32:47.646544 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:47.646516 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gbc4r" event={"ID":"3e35ffc8-ef10-4532-af91-38841c3f4581","Type":"ContainerStarted","Data":"dcc69eae5c7774f8b56c3f671e21766dce7bf983a0337b36318d9b06cc76d13d"} Nov 02 17:32:47.835791 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:47.835699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:32:47.835971 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:47.835826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tmr\" (UniqueName: \"kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr\") pod \"network-check-target-424d5\" (UID: \"9ce34815-aef2-487a-afd1-fe81200a423d\") " pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:32:47.836040 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:47.835976 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:32:47.836040 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:47.836000 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:32:47.836040 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:47.836014 2579 projected.go:196] Error preparing data for projected volume kube-api-access-m9tmr for pod openshift-network-diagnostics/network-check-target-424d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:47.836199 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:47.836074 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr podName:9ce34815-aef2-487a-afd1-fe81200a423d nodeName:}" failed. No retries permitted until 2026-11-02 17:32:49.836056175 +0000 UTC m=+5.049629609 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9tmr" (UniqueName: "kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr") pod "network-check-target-424d5" (UID: "9ce34815-aef2-487a-afd1-fe81200a423d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:47.836642 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:47.836520 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:47.836642 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:47.836576 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret podName:d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:49.836561081 +0000 UTC m=+5.050134509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret") pod "global-pull-secret-syncer-f4xwb" (UID: "d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:47.937270 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:47.937220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs\") pod \"network-metrics-daemon-dfthz\" (UID: \"86286bdb-d69d-481a-b874-4ff3df42906f\") " pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:32:47.937469 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:47.937427 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:47.937543 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:47.937498 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs podName:86286bdb-d69d-481a-b874-4ff3df42906f nodeName:}" failed. No retries permitted until 2026-11-02 17:32:49.937476958 +0000 UTC m=+5.151050374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs") pod "network-metrics-daemon-dfthz" (UID: "86286bdb-d69d-481a-b874-4ff3df42906f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:48.261371 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:48.261242 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:27:47 +0000 UTC" deadline="2028-07-04 01:02:52.083034517 +0000 UTC" Nov 02 17:32:48.261371 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:48.261304 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14623h30m3.821734675s" Nov 02 17:32:48.483767 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:48.483729 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:32:48.483962 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:48.483878 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:32:48.484382 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:48.484361 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:32:48.484498 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:48.484474 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:32:48.484562 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:48.484541 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:32:48.484626 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:48.484610 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:32:49.862105 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:49.861925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tmr\" (UniqueName: \"kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr\") pod \"network-check-target-424d5\" (UID: \"9ce34815-aef2-487a-afd1-fe81200a423d\") " pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:32:49.862105 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:49.861986 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:32:49.862671 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:49.862174 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:49.862671 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:49.862237 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret podName:d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:53.862217648 +0000 UTC m=+9.075791062 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret") pod "global-pull-secret-syncer-f4xwb" (UID: "d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:49.862671 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:49.862668 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:32:49.862869 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:49.862688 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:32:49.862869 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:49.862715 2579 projected.go:196] Error preparing data for projected volume kube-api-access-m9tmr for pod openshift-network-diagnostics/network-check-target-424d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:49.862869 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:49.862763 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr podName:9ce34815-aef2-487a-afd1-fe81200a423d nodeName:}" failed. No retries permitted until 2026-11-02 17:32:53.862747509 +0000 UTC m=+9.076320940 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9tmr" (UniqueName: "kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr") pod "network-check-target-424d5" (UID: "9ce34815-aef2-487a-afd1-fe81200a423d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:49.963217 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:49.963177 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs\") pod \"network-metrics-daemon-dfthz\" (UID: \"86286bdb-d69d-481a-b874-4ff3df42906f\") " pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:32:49.963414 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:49.963366 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:49.963483 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:49.963433 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs podName:86286bdb-d69d-481a-b874-4ff3df42906f nodeName:}" failed. No retries permitted until 2026-11-02 17:32:53.96341271 +0000 UTC m=+9.176986137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs") pod "network-metrics-daemon-dfthz" (UID: "86286bdb-d69d-481a-b874-4ff3df42906f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:50.484052 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:50.483964 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:32:50.484223 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:50.484124 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:32:50.484567 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:50.484543 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:32:50.484716 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:50.484669 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:32:50.484788 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:50.484745 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:32:50.484844 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:50.484824 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:32:51.692222 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:51.691708 2579 generic.go:356] "Generic (PLEG): container finished" podID="8d225fb4-f477-4971-bce1-8aaa24a79be8" containerID="8f1b6613840c3b99479194e69f1c0929c4f066d5d03d12f80f650d44b1cabebd" exitCode=0 Nov 02 17:32:51.692222 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:51.692036 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdll2" event={"ID":"8d225fb4-f477-4971-bce1-8aaa24a79be8","Type":"ContainerDied","Data":"8f1b6613840c3b99479194e69f1c0929c4f066d5d03d12f80f650d44b1cabebd"} Nov 02 17:32:51.701734 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:51.701180 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cjnbh" event={"ID":"c00d16e7-b92c-46c7-a68f-797274bbef93","Type":"ContainerStarted","Data":"c8f8de3e614027eea7ce94e63bc11e25c413bbcf489dd177e8c2bdf0ff85ad35"} Nov 02 17:32:51.704981 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:51.704330 2579 generic.go:356] "Generic (PLEG): container finished" podID="ca96163b2bc1e65e3d055163a6272914" containerID="3d5c1447bb3db591bd57f2cb5192f39f60f495a403bbfa9aa442646e744abaf3" exitCode=0 Nov 02 17:32:51.704981 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:51.704373 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" event={"ID":"ca96163b2bc1e65e3d055163a6272914","Type":"ContainerDied","Data":"3d5c1447bb3db591bd57f2cb5192f39f60f495a403bbfa9aa442646e744abaf3"} Nov 02 17:32:51.734779 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:51.734711 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cjnbh" podStartSLOduration=0.973227837 podStartE2EDuration="5.734672455s" totalImagesPullingTime="4.761444618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:51.734365824 +0000 UTC m=+6.947939261" watchObservedRunningTime="2026-11-02 17:32:51.734672455 +0000 UTC m=+6.948245891" Nov 02 17:32:52.232820 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:52.232739 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cjnbh" Nov 02 17:32:52.484281 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:52.484204 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:32:52.484452 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:52.484345 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:32:52.484814 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:52.484794 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:32:52.484941 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:52.484916 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:32:52.485002 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:52.484993 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:32:52.485085 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:52.485063 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:32:52.716128 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:52.716086 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" event={"ID":"ca96163b2bc1e65e3d055163a6272914","Type":"ContainerStarted","Data":"e98b36ae1bfaf2e9af9278394c6aae99ae76ed9d74424268d5a56c599de5ddd2"} Nov 02 17:32:52.723888 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:52.723859 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal" event={"ID":"f82e5ebcf2f8d3b5e95e3af83d2365db","Type":"ContainerStarted","Data":"2f70519c3749ca0c14bcc25d82701ef9b60acbaeb2b43bb7ff828d99fd572d32"} Nov 02 17:32:52.750743 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:52.750624 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" podStartSLOduration=6.750600332 podStartE2EDuration="6.750600332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:52.735113207 +0000 UTC m=+7.948686644" watchObservedRunningTime="2026-11-02 17:32:52.750600332 +0000 UTC m=+7.964173770" Nov 02 17:32:53.735585 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:53.735544 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q28kf" event={"ID":"d9300fb6-6926-4429-8797-603dfb361818","Type":"ContainerStarted","Data":"232279fddbdd4d63025d015ad606dfa64458dc3e1cb4de38d5535df912908545"} Nov 02 17:32:53.752843 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:53.752768 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q28kf" podStartSLOduration=1.740114095 podStartE2EDuration="7.752746941s" totalImagesPullingTime="6.012632846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:53.752215669 +0000 UTC m=+8.965789106" watchObservedRunningTime="2026-11-02 17:32:53.752746941 +0000 UTC m=+8.966320377" Nov 02 17:32:53.753124 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:53.753091 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal" podStartSLOduration=7.753080052 podStartE2EDuration="7.753080052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:52.751475235 +0000 UTC m=+7.965048673" watchObservedRunningTime="2026-11-02 17:32:53.753080052 +0000 UTC m=+8.966653493" Nov 02 17:32:53.899848 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:53.898910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tmr\" (UniqueName: \"kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr\") pod \"network-check-target-424d5\" (UID: \"9ce34815-aef2-487a-afd1-fe81200a423d\") " pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:32:53.899848 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:53.898981 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:32:53.899848 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:53.899148 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:53.899848 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:53.899235 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret podName:d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:01.899193742 +0000 UTC m=+17.112767157 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret") pod "global-pull-secret-syncer-f4xwb" (UID: "d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:32:53.899848 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:53.899651 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:32:53.899848 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:53.899667 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:32:53.899848 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:53.899683 2579 projected.go:196] Error preparing data for projected volume kube-api-access-m9tmr for pod openshift-network-diagnostics/network-check-target-424d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:53.899848 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:53.899774 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr podName:9ce34815-aef2-487a-afd1-fe81200a423d nodeName:}" failed. No retries permitted until 2026-11-02 17:33:01.899758336 +0000 UTC m=+17.113331754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9tmr" (UniqueName: "kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr") pod "network-check-target-424d5" (UID: "9ce34815-aef2-487a-afd1-fe81200a423d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:32:54.000404 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:53.999715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs\") pod \"network-metrics-daemon-dfthz\" (UID: \"86286bdb-d69d-481a-b874-4ff3df42906f\") " pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:32:54.000404 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:53.999890 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:54.000404 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:53.999980 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs podName:86286bdb-d69d-481a-b874-4ff3df42906f nodeName:}" failed. No retries permitted until 2026-11-02 17:33:01.999960414 +0000 UTC m=+17.213533828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs") pod "network-metrics-daemon-dfthz" (UID: "86286bdb-d69d-481a-b874-4ff3df42906f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:32:54.484764 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:54.484002 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:32:54.484764 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:54.484131 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:32:54.484764 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:54.484456 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:32:54.484764 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:54.484562 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:32:54.484764 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:54.484626 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:32:54.484764 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:54.484730 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:32:54.739692 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:54.739602 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" event={"ID":"40e6a790-de24-49f1-b507-ed74d1446868","Type":"ContainerStarted","Data":"297c5a8698ae7633d11cb7541c36d865bdec090c28617fb9bb7474c317352406"} Nov 02 17:32:55.662719 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:55.662670 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 17:32:55.751454 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:55.751416 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" event={"ID":"40e6a790-de24-49f1-b507-ed74d1446868","Type":"ContainerStarted","Data":"90129fe7e8228c41d93c918b80b820bc211c9f2a48d27a8b8f047389a445b50e"} Nov 02 17:32:55.753820 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:55.753784 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gbc4r" event={"ID":"3e35ffc8-ef10-4532-af91-38841c3f4581","Type":"ContainerStarted","Data":"097020a249fa466ab9b2119e8774d0166eb06ebd21a3018712d83de9e82d3d47"} Nov 02 17:32:55.773575 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:55.773517 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-gbc4r" podStartSLOduration=1.175047315 podStartE2EDuration="9.773495554s" totalImagesPullingTime="8.598448239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:55.773354132 +0000 UTC m=+10.986927569" watchObservedRunningTime="2026-11-02 17:32:55.773495554 +0000 UTC m=+10.987068994" Nov 02 17:32:56.131022 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:56.130933 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cjnbh" Nov 02 17:32:56.131632 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:56.131610 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cjnbh" Nov 02 17:32:56.274359 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:56.274247 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:32:55.662697375Z","UUID":"22541398-910e-4e81-91b2-2995810ad67e","Handler":null,"Name":"","Endpoint":""} Nov 02 17:32:56.275926 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:56.275873 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 17:32:56.275926 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:56.275905 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 17:32:56.483994 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:56.483959 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:32:56.484167 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:56.483994 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:32:56.484167 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:56.484103 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:32:56.484286 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:56.484218 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:32:56.484286 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:56.484224 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:32:56.484381 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:56.484319 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:32:56.760208 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:56.760122 2579 generic.go:356] "Generic (PLEG): container finished" podID="8d225fb4-f477-4971-bce1-8aaa24a79be8" containerID="e82ced2864ae60849035add8cd5469380c1f855ab50b79038a475e5fd992c184" exitCode=0 Nov 02 17:32:56.760811 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:56.760221 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdll2" event={"ID":"8d225fb4-f477-4971-bce1-8aaa24a79be8","Type":"ContainerDied","Data":"e82ced2864ae60849035add8cd5469380c1f855ab50b79038a475e5fd992c184"} Nov 02 17:32:56.763650 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:56.763563 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5tkc8" event={"ID":"a48715bf-e5aa-4b50-a9a5-4916e6fbefbc","Type":"ContainerStarted","Data":"064905e51946c324e0ef1887a6f4fe4208c9c87a4888e4b9944e05fbc0fce323"} Nov 02 17:32:56.765331 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:56.765311 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cjnbh" Nov 02 17:32:56.831691 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:56.831108 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5tkc8" podStartSLOduration=2.047238396 podStartE2EDuration="10.83108773s" totalImagesPullingTime="8.783849334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:56.830488118 +0000 UTC m=+12.044061553" watchObservedRunningTime="2026-11-02 17:32:56.83108773 +0000 UTC m=+12.044661166" Nov 02 17:32:57.768455 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:57.768325 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5l7cq" event={"ID":"44174ea2-6704-423f-a176-c3abe64be85e","Type":"ContainerStarted","Data":"7f369bcd39b6865cddbcb7bc4fa904b92d8bf032884e086db6202dae58fcaa9b"} Nov 02 17:32:57.771557 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:57.771524 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" event={"ID":"40e6a790-de24-49f1-b507-ed74d1446868","Type":"ContainerStarted","Data":"530c9f7ce6458aee0d8091c0bcfb2e00ac8eb0eda3769549e4a8eea99769bb79"} Nov 02 17:32:57.774577 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:57.774120 2579 generic.go:356] "Generic (PLEG): container finished" podID="8d225fb4-f477-4971-bce1-8aaa24a79be8" containerID="14846034d093b9c9eaa4f86453671cb7514e2858b0960aa6afb3f48a5e2c211e" exitCode=0 Nov 02 17:32:57.774577 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:57.774228 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdll2" event={"ID":"8d225fb4-f477-4971-bce1-8aaa24a79be8","Type":"ContainerDied","Data":"14846034d093b9c9eaa4f86453671cb7514e2858b0960aa6afb3f48a5e2c211e"} Nov 02 17:32:57.787550 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:57.787507 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5l7cq" podStartSLOduration=2.3122448 podStartE2EDuration="11.787492926s" totalImagesPullingTime="9.475248126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:57.786658441 +0000 UTC m=+13.000231877" watchObservedRunningTime="2026-11-02 17:32:57.787492926 +0000 UTC m=+13.001066361" Nov 02 17:32:57.829831 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:57.829774 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" podStartSLOduration=2.056917932 podStartE2EDuration="11.829754711s" totalImagesPullingTime="9.772836779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:57.82894574 +0000 UTC m=+13.042519175" watchObservedRunningTime="2026-11-02 17:32:57.829754711 +0000 UTC m=+13.043328148" Nov 02 17:32:58.483443 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:58.483414 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:32:58.483580 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:58.483427 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:32:58.483580 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:58.483528 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:32:58.483580 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:58.483524 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:32:58.483760 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:58.483593 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:32:58.483760 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:32:58.483654 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:32:58.779549 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:58.779345 2579 generic.go:356] "Generic (PLEG): container finished" podID="8d225fb4-f477-4971-bce1-8aaa24a79be8" containerID="0ddbc79f333f6590139d6461f85e401600eb42b4c7a2f9a7b15e57fcab327e5a" exitCode=0 Nov 02 17:32:58.780209 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:58.779426 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdll2" event={"ID":"8d225fb4-f477-4971-bce1-8aaa24a79be8","Type":"ContainerDied","Data":"0ddbc79f333f6590139d6461f85e401600eb42b4c7a2f9a7b15e57fcab327e5a"} Nov 02 17:32:58.781282 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:58.781232 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" event={"ID":"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e","Type":"ContainerStarted","Data":"9209bd0721889df6aebb7d83c4ff14c24ec9f5d48a3832c4dbcfa5d1836433e3"} Nov 02 17:32:58.824628 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:32:58.824571 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" podStartSLOduration=1.08880801 podStartE2EDuration="12.824552937s" totalImagesPullingTime="11.735744927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:58.8240843 +0000 UTC m=+14.037657734" watchObservedRunningTime="2026-11-02 17:32:58.824552937 +0000 UTC m=+14.038126368" Nov 02 17:33:00.483698 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:00.483665 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:00.484184 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:00.483732 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:00.484184 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:00.483865 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:33:00.484321 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:00.484238 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:00.484369 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:00.484341 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:33:00.484445 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:00.484421 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:33:00.789893 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:00.789806 2579 generic.go:356] "Generic (PLEG): container finished" podID="8d225fb4-f477-4971-bce1-8aaa24a79be8" containerID="77819e1f88605c92d8fda1bf242fba510f367e34da769b2d6b921d7b94f30270" exitCode=0 Nov 02 17:33:00.790041 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:00.789882 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdll2" event={"ID":"8d225fb4-f477-4971-bce1-8aaa24a79be8","Type":"ContainerDied","Data":"77819e1f88605c92d8fda1bf242fba510f367e34da769b2d6b921d7b94f30270"} Nov 02 17:33:01.795809 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:01.795774 2579 generic.go:356] "Generic (PLEG): container finished" podID="8d225fb4-f477-4971-bce1-8aaa24a79be8" containerID="3c5d77d76c80b259b3a53fb8e0951965ee0149607f86de4d2b355a2aa64caf41" exitCode=0 Nov 02 17:33:01.796284 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:01.795823 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdll2" event={"ID":"8d225fb4-f477-4971-bce1-8aaa24a79be8","Type":"ContainerDied","Data":"3c5d77d76c80b259b3a53fb8e0951965ee0149607f86de4d2b355a2aa64caf41"} Nov 02 17:33:01.968218 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:01.968186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tmr\" (UniqueName: \"kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr\") pod \"network-check-target-424d5\" (UID: \"9ce34815-aef2-487a-afd1-fe81200a423d\") " pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:01.968218 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:01.968223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:01.968469 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:01.968345 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:33:01.968469 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:01.968397 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret podName:d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:17.968383944 +0000 UTC m=+33.181957355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret") pod "global-pull-secret-syncer-f4xwb" (UID: "d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:33:01.968469 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:01.968349 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:33:01.968469 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:01.968442 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:33:01.968469 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:01.968455 2579 projected.go:196] Error preparing data for projected volume kube-api-access-m9tmr for pod openshift-network-diagnostics/network-check-target-424d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:33:01.968642 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:01.968494 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr podName:9ce34815-aef2-487a-afd1-fe81200a423d nodeName:}" failed. No retries permitted until 2026-11-02 17:33:17.968480855 +0000 UTC m=+33.182054284 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9tmr" (UniqueName: "kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr") pod "network-check-target-424d5" (UID: "9ce34815-aef2-487a-afd1-fe81200a423d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:33:02.072009 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:02.069443 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs\") pod \"network-metrics-daemon-dfthz\" (UID: \"86286bdb-d69d-481a-b874-4ff3df42906f\") " pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:02.072009 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:02.069724 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:33:02.072009 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:02.069805 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs podName:86286bdb-d69d-481a-b874-4ff3df42906f nodeName:}" failed. No retries permitted until 2026-11-02 17:33:18.069784597 +0000 UTC m=+33.283358024 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs") pod "network-metrics-daemon-dfthz" (UID: "86286bdb-d69d-481a-b874-4ff3df42906f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:33:02.484109 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:02.484028 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:02.484109 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:02.484052 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:02.484109 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:02.484069 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:02.484365 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:02.484180 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:33:02.484365 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:02.484292 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:33:02.484365 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:02.484359 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:33:02.802264 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:02.801787 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdll2" event={"ID":"8d225fb4-f477-4971-bce1-8aaa24a79be8","Type":"ContainerStarted","Data":"160f00c4973974421c6b0479521011b6015f5cdfbf45472b95f821460831bf5b"} Nov 02 17:33:02.804015 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:02.803992 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" event={"ID":"11bb6b24-b4e9-4764-a40a-e1daa2c613c2","Type":"ContainerStarted","Data":"4b675416a7b009d4fcd263d419ae901395522ce9c3beb40f7cf28e4a57cd95cd"} Nov 02 17:33:02.804113 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:02.804057 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" event={"ID":"11bb6b24-b4e9-4764-a40a-e1daa2c613c2","Type":"ContainerStarted","Data":"959dcb3a3ecd589ebf985920e4425db2c6e252f2d79c389b3219d017a1038ae2"} Nov 02 17:33:02.832756 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:02.832710 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vdll2" podStartSLOduration=5.948290237 podStartE2EDuration="16.832697332s" totalImagesPullingTime="10.884407095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:02.832380916 +0000 UTC m=+18.045954362" watchObservedRunningTime="2026-11-02 17:33:02.832697332 +0000 UTC m=+18.046270764" Nov 02 17:33:03.808632 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:03.808584 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" event={"ID":"11bb6b24-b4e9-4764-a40a-e1daa2c613c2","Type":"ContainerStarted","Data":"7a869165445669d6b3a7071330e0de23471f12e1f8b388152e8e4902473be334"} Nov 02 17:33:03.809093 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:03.808688 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" event={"ID":"11bb6b24-b4e9-4764-a40a-e1daa2c613c2","Type":"ContainerStarted","Data":"2ade308f0d2ec37edca33e03f05ed199b1ed8fcf3a67b1a25dd23c21264f4eaf"} Nov 02 17:33:03.809093 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:03.808741 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" event={"ID":"11bb6b24-b4e9-4764-a40a-e1daa2c613c2","Type":"ContainerStarted","Data":"d88ecc90b319677af93019b6ce302ec5353c3d8e795ddde2b3ec4d0a60940382"} Nov 02 17:33:03.809093 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:03.808789 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" event={"ID":"11bb6b24-b4e9-4764-a40a-e1daa2c613c2","Type":"ContainerStarted","Data":"5325ab4f39acd37f1406b666386a390d3f65bb36c25075db1827b3b6f60fe670"} Nov 02 17:33:04.483419 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:04.483357 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:04.483663 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:04.483372 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:04.483663 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:04.483508 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:33:04.483663 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:04.483357 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:04.483663 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:04.483599 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:33:04.483663 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:04.483663 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:33:05.815269 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:05.815232 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" event={"ID":"11bb6b24-b4e9-4764-a40a-e1daa2c613c2","Type":"ContainerStarted","Data":"075f4c476e09d89031d269b7e7c011e2c2d18331cf4c81fd3196977553f281dc"} Nov 02 17:33:06.483947 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:06.483909 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:06.484157 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:06.483916 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:06.484157 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:06.484038 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:33:06.484157 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:06.483916 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:06.484157 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:06.484098 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:33:06.484352 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:06.484170 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:33:07.823745 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:07.823703 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" event={"ID":"11bb6b24-b4e9-4764-a40a-e1daa2c613c2","Type":"ContainerStarted","Data":"b5a79e385a9250a1f406cdca3c30ac60aa6771aefef67702f0fd75c047fa0b8a"} Nov 02 17:33:07.824194 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:07.823945 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:07.839029 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:07.839008 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:07.856432 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:07.856392 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" podStartSLOduration=5.871339293 podStartE2EDuration="21.856381763s" totalImagesPullingTime="15.98504247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:07.8546043 +0000 UTC m=+23.068177742" watchObservedRunningTime="2026-11-02 17:33:07.856381763 +0000 UTC m=+23.069955194" Nov 02 17:33:08.483231 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:08.483197 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:08.483231 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:08.483218 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:08.483527 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:08.483199 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:08.483527 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:08.483314 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:33:08.483527 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:08.483361 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:33:08.483527 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:08.483474 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:33:08.826225 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:08.826148 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:33:08.826647 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:08.826582 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:08.841658 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:08.841637 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:09.267965 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:09.267941 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5tkc8_a48715bf-e5aa-4b50-a9a5-4916e6fbefbc/dns-node-resolver/0.log" Nov 02 17:33:09.828747 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:09.828720 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:33:10.452270 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:10.452239 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q28kf_d9300fb6-6926-4429-8797-603dfb361818/node-ca/0.log" Nov 02 17:33:10.483785 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:10.483753 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:10.483887 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:10.483870 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:10.483976 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:10.483959 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:33:10.484036 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:10.483872 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:33:10.484036 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:10.483877 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:10.484036 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:10.484028 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:33:10.830943 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:10.830865 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:33:12.483463 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:12.483421 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:12.484017 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:12.483421 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:12.484017 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:12.483551 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:33:12.484017 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:12.483608 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:33:12.484017 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:12.483432 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:12.484017 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:12.483701 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:33:14.484093 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:14.484058 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:14.484524 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:14.484058 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:14.484524 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:14.484169 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:33:14.484524 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:14.484058 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:14.484524 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:14.484250 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:33:14.484524 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:14.484295 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:33:15.909556 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:15.909508 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:15.909976 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:15.909792 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:33:15.922370 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:15.922327 2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" podUID="11bb6b24-b4e9-4764-a40a-e1daa2c613c2" containerName="ovnkube-controller" probeResult="failure" output="" Nov 02 17:33:15.930507 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:15.930482 2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" podUID="11bb6b24-b4e9-4764-a40a-e1daa2c613c2" containerName="ovnkube-controller" probeResult="failure" output="" Nov 02 17:33:16.483997 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:16.483958 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:16.484167 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:16.484021 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:16.484167 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:16.484103 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:16.484167 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:16.484115 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:33:16.484313 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:16.484176 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:33:16.484313 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:16.484228 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:33:17.997381 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:17.997354 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tmr\" (UniqueName: \"kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr\") pod \"network-check-target-424d5\" (UID: \"9ce34815-aef2-487a-afd1-fe81200a423d\") " pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:17.997381 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:17.997385 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:17.997828 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:17.997484 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:33:17.997828 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:17.997498 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:33:17.997828 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:17.997514 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:33:17.997828 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:17.997525 2579 projected.go:196] Error preparing data for projected volume kube-api-access-m9tmr for pod openshift-network-diagnostics/network-check-target-424d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:33:17.997828 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:17.997545 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret podName:d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:49.997527193 +0000 UTC m=+65.211100608 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret") pod "global-pull-secret-syncer-f4xwb" (UID: "d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:33:17.997828 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:17.997574 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr podName:9ce34815-aef2-487a-afd1-fe81200a423d nodeName:}" failed. No retries permitted until 2026-11-02 17:33:49.997564952 +0000 UTC m=+65.211138379 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9tmr" (UniqueName: "kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr") pod "network-check-target-424d5" (UID: "9ce34815-aef2-487a-afd1-fe81200a423d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:33:18.097573 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:18.097545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs\") pod \"network-metrics-daemon-dfthz\" (UID: \"86286bdb-d69d-481a-b874-4ff3df42906f\") " pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:18.097769 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:18.097753 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:33:18.097880 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:18.097818 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs podName:86286bdb-d69d-481a-b874-4ff3df42906f nodeName:}" failed. No retries permitted until 2026-11-02 17:33:50.097800123 +0000 UTC m=+65.311373547 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs") pod "network-metrics-daemon-dfthz" (UID: "86286bdb-d69d-481a-b874-4ff3df42906f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:33:18.483512 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:18.483487 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:18.483709 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:18.483603 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:33:18.483709 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:18.483645 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:18.483709 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:18.483685 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:18.483886 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:18.483775 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:33:18.483886 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:18.483864 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:33:20.483177 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:20.483144 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:20.483681 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:20.483144 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:20.483681 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:20.483256 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:33:20.483681 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:20.483147 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:20.483681 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:20.483319 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:33:20.483681 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:20.483374 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:33:22.483625 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:22.483595 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:22.484019 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:22.483636 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:22.484019 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:22.483696 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:22.484019 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:22.483708 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:33:22.484019 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:22.483772 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:33:22.484019 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:22.483826 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:33:24.484047 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:24.484014 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:24.484496 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:24.484027 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:24.484496 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:24.484136 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:33:24.484496 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:24.484033 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:24.484496 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:24.484205 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:33:24.484496 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:24.484311 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:33:25.857444 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:25.857216 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dfthz"] Nov 02 17:33:25.857947 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:25.857525 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:25.857947 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:25.857675 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:33:25.861783 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:25.861751 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f4xwb"] Nov 02 17:33:25.861924 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:25.861867 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:25.861985 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:25.861960 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:33:25.862387 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:25.862367 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-424d5"] Nov 02 17:33:25.862450 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:25.862441 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:25.862513 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:25.862498 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:33:27.483229 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:27.483201 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:27.483726 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:27.483207 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:27.483726 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:27.483207 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:27.483726 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:27.483349 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:33:27.483726 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:27.483383 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:33:27.483726 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:27.483444 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:33:29.484112 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:29.484071 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:29.484820 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:29.484142 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:29.484820 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:29.484154 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:29.484820 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:29.484229 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-424d5" podUID="9ce34815-aef2-487a-afd1-fe81200a423d" Nov 02 17:33:29.484820 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:29.484320 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfthz" podUID="86286bdb-d69d-481a-b874-4ff3df42906f" Nov 02 17:33:29.484820 ip-10-0-141-169 kubenswrapper[2579]: E1102 17:33:29.484384 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4xwb" podUID="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" Nov 02 17:33:30.073551 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.073527 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-169.us-east-2.compute.internal" event="NodeReady" Nov 02 17:33:30.073739 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.073682 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 17:33:30.123376 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.123349 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rtxf5"] Nov 02 17:33:30.125387 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.125363 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:30.129810 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.129789 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vrm5q"] Nov 02 17:33:30.131850 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.131828 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:30.132804 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.132786 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dp4wq\"" Nov 02 17:33:30.132883 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.132786 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 17:33:30.132883 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.132845 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 17:33:30.136637 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.136602 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rtxf5"] Nov 02 17:33:30.147884 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.147867 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vrm5q"] Nov 02 17:33:30.153934 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.153914 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nwbhl"] Nov 02 17:33:30.155686 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.155669 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nwbhl" Nov 02 17:33:30.166774 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.166753 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nwbhl"] Nov 02 17:33:30.234824 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.234745 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 17:33:30.234824 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.234780 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vxnnv\"" Nov 02 17:33:30.234824 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.234779 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 17:33:30.235743 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.235729 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 17:33:30.236004 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.235990 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 17:33:30.236042 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.235994 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 17:33:30.236042 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.235994 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 17:33:30.236266 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.236255 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9wn4g\"" Nov 02 17:33:30.236302 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.236258 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 17:33:30.290673 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.290650 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdccp\" (UniqueName: \"kubernetes.io/projected/afbca029-7d59-4cfb-b08c-353e174b0355-kube-api-access-hdccp\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:30.290785 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.290677 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9372be33-753d-4006-ba16-86fc44bd7c52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:30.290785 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.290691 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/afbca029-7d59-4cfb-b08c-353e174b0355-config-volume\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:30.290785 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.290716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9372be33-753d-4006-ba16-86fc44bd7c52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:30.290897 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.290818 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afbca029-7d59-4cfb-b08c-353e174b0355-metrics-tls\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:30.290897 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.290842 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9372be33-753d-4006-ba16-86fc44bd7c52-crio-socket\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:30.290897 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.290858 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afbca029-7d59-4cfb-b08c-353e174b0355-tmp-dir\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:30.290897 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.290875 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vjcv\" (UniqueName: \"kubernetes.io/projected/a47b32de-3a53-4b34-8874-87049c4d747b-kube-api-access-7vjcv\") pod \"ingress-canary-nwbhl\" (UID: \"a47b32de-3a53-4b34-8874-87049c4d747b\") " pod="openshift-ingress-canary/ingress-canary-nwbhl" Nov 02 17:33:30.291031 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.290971 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9372be33-753d-4006-ba16-86fc44bd7c52-data-volume\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:30.291031 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.290994 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpg54\" (UniqueName: \"kubernetes.io/projected/9372be33-753d-4006-ba16-86fc44bd7c52-kube-api-access-rpg54\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:30.291031 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.291010 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a47b32de-3a53-4b34-8874-87049c4d747b-cert\") pod \"ingress-canary-nwbhl\" (UID: \"a47b32de-3a53-4b34-8874-87049c4d747b\") " pod="openshift-ingress-canary/ingress-canary-nwbhl" Nov 02 17:33:30.392109 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.392084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdccp\" (UniqueName: \"kubernetes.io/projected/afbca029-7d59-4cfb-b08c-353e174b0355-kube-api-access-hdccp\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:30.392219 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.392112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9372be33-753d-4006-ba16-86fc44bd7c52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:30.392219 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.392138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/afbca029-7d59-4cfb-b08c-353e174b0355-config-volume\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:30.392219 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.392163 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9372be33-753d-4006-ba16-86fc44bd7c52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:30.392219 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.392198 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afbca029-7d59-4cfb-b08c-353e174b0355-metrics-tls\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:30.392219 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.392221 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9372be33-753d-4006-ba16-86fc44bd7c52-crio-socket\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:30.392458 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.392235 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afbca029-7d59-4cfb-b08c-353e174b0355-tmp-dir\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:30.392458 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.392249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vjcv\" (UniqueName: \"kubernetes.io/projected/a47b32de-3a53-4b34-8874-87049c4d747b-kube-api-access-7vjcv\") pod \"ingress-canary-nwbhl\" (UID: \"a47b32de-3a53-4b34-8874-87049c4d747b\") " pod="openshift-ingress-canary/ingress-canary-nwbhl" Nov 02 17:33:30.392458 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.392288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9372be33-753d-4006-ba16-86fc44bd7c52-data-volume\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:30.392458 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.392311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpg54\" (UniqueName: \"kubernetes.io/projected/9372be33-753d-4006-ba16-86fc44bd7c52-kube-api-access-rpg54\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:30.392458 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.392334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a47b32de-3a53-4b34-8874-87049c4d747b-cert\") pod \"ingress-canary-nwbhl\" (UID: \"a47b32de-3a53-4b34-8874-87049c4d747b\") " pod="openshift-ingress-canary/ingress-canary-nwbhl" Nov 02 17:33:30.392725 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.392488 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9372be33-753d-4006-ba16-86fc44bd7c52-crio-socket\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:30.392953 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.392792 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9372be33-753d-4006-ba16-86fc44bd7c52-data-volume\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:30.392953 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.392799 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afbca029-7d59-4cfb-b08c-353e174b0355-tmp-dir\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:30.392953 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.392873 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9372be33-753d-4006-ba16-86fc44bd7c52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:30.392953 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.392906 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/afbca029-7d59-4cfb-b08c-353e174b0355-config-volume\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:30.396219 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.396201 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afbca029-7d59-4cfb-b08c-353e174b0355-metrics-tls\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:30.396305 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.396250 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9372be33-753d-4006-ba16-86fc44bd7c52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:30.396387 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.396372 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a47b32de-3a53-4b34-8874-87049c4d747b-cert\") pod \"ingress-canary-nwbhl\" (UID: \"a47b32de-3a53-4b34-8874-87049c4d747b\") " pod="openshift-ingress-canary/ingress-canary-nwbhl" Nov 02 17:33:30.401703 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.401681 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdccp\" (UniqueName: \"kubernetes.io/projected/afbca029-7d59-4cfb-b08c-353e174b0355-kube-api-access-hdccp\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:30.401868 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.401847 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpg54\" (UniqueName: \"kubernetes.io/projected/9372be33-753d-4006-ba16-86fc44bd7c52-kube-api-access-rpg54\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:30.401955 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.401941 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vjcv\" (UniqueName: \"kubernetes.io/projected/a47b32de-3a53-4b34-8874-87049c4d747b-kube-api-access-7vjcv\") pod \"ingress-canary-nwbhl\" (UID: \"a47b32de-3a53-4b34-8874-87049c4d747b\") " pod="openshift-ingress-canary/ingress-canary-nwbhl" Nov 02 17:33:30.437650 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.437626 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:30.445836 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.445816 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:30.465200 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.465175 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nwbhl" Nov 02 17:33:30.723409 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.723380 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rtxf5"] Nov 02 17:33:30.727099 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:33:30.727064 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafbca029_7d59_4cfb_b08c_353e174b0355.slice/crio-2771f15ce74dbd7bbb1485be9914a1064a4f50bdf3e0552ffb65daab89654c24 WatchSource:0}: Error finding container 2771f15ce74dbd7bbb1485be9914a1064a4f50bdf3e0552ffb65daab89654c24: Status 404 returned error can't find the container with id 2771f15ce74dbd7bbb1485be9914a1064a4f50bdf3e0552ffb65daab89654c24 Nov 02 17:33:30.727527 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.727507 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vrm5q"] Nov 02 17:33:30.729996 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:33:30.729976 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9372be33_753d_4006_ba16_86fc44bd7c52.slice/crio-73e7fa334d83301b9af102efa79afce3f99e00e42080633b168bbcd8a22c9fee WatchSource:0}: Error finding container 73e7fa334d83301b9af102efa79afce3f99e00e42080633b168bbcd8a22c9fee: Status 404 returned error can't find the container with id 73e7fa334d83301b9af102efa79afce3f99e00e42080633b168bbcd8a22c9fee Nov 02 17:33:30.757021 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.756488 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nwbhl"] Nov 02 17:33:30.762187 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:33:30.762159 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47b32de_3a53_4b34_8874_87049c4d747b.slice/crio-077f6bade89779397e191fbf68451bda0f46f291e1b2a9f81bfee8688a9d1c07 WatchSource:0}: Error finding container 077f6bade89779397e191fbf68451bda0f46f291e1b2a9f81bfee8688a9d1c07: Status 404 returned error can't find the container with id 077f6bade89779397e191fbf68451bda0f46f291e1b2a9f81bfee8688a9d1c07 Nov 02 17:33:30.872399 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.872363 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rtxf5" event={"ID":"afbca029-7d59-4cfb-b08c-353e174b0355","Type":"ContainerStarted","Data":"2771f15ce74dbd7bbb1485be9914a1064a4f50bdf3e0552ffb65daab89654c24"} Nov 02 17:33:30.873404 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.873381 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nwbhl" event={"ID":"a47b32de-3a53-4b34-8874-87049c4d747b","Type":"ContainerStarted","Data":"077f6bade89779397e191fbf68451bda0f46f291e1b2a9f81bfee8688a9d1c07"} Nov 02 17:33:30.874747 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.874727 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vrm5q" event={"ID":"9372be33-753d-4006-ba16-86fc44bd7c52","Type":"ContainerStarted","Data":"6a35ee4a404c218c0b56ca5985a8582b0eacae52f0b8a77ef29521d9d0be0b92"} Nov 02 17:33:30.874833 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:30.874771 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vrm5q" event={"ID":"9372be33-753d-4006-ba16-86fc44bd7c52","Type":"ContainerStarted","Data":"73e7fa334d83301b9af102efa79afce3f99e00e42080633b168bbcd8a22c9fee"} Nov 02 17:33:31.484400 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.484171 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:31.485232 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.484924 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:31.485360 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.485328 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:31.490434 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.490415 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 17:33:31.490781 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.490761 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dcw65\"" Nov 02 17:33:31.490890 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.490767 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 17:33:31.490890 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.490828 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 17:33:31.490890 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.490835 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jmxfr\"" Nov 02 17:33:31.491016 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.490921 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-52q82\"" Nov 02 17:33:31.491066 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.491051 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 17:33:31.872212 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.871113 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph"] Nov 02 17:33:31.873388 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.873363 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:31.883871 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.883846 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vrm5q" event={"ID":"9372be33-753d-4006-ba16-86fc44bd7c52","Type":"ContainerStarted","Data":"b9a46b8eadc0daedc0f86e3390d20f4549f664cfb11465abb64608910482c657"} Nov 02 17:33:31.884092 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.884074 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 17:33:31.884182 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.884160 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 17:33:31.884294 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.884259 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 17:33:31.884492 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.884473 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jdltr\"" Nov 02 17:33:31.884641 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.884609 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 17:33:31.884839 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.884821 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 17:33:31.891369 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.890018 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph"] Nov 02 17:33:31.907386 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.907363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:31.907494 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.907400 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477"] Nov 02 17:33:31.907494 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.907413 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqn2p\" (UniqueName: \"kubernetes.io/projected/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-kube-api-access-pqn2p\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:31.907494 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.907444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:31.907494 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.907473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:31.910944 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.910009 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:31.910944 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.910152 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8dtcc"] Nov 02 17:33:31.913121 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.913103 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:31.923737 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.923684 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477"] Nov 02 17:33:31.937273 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.937233 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wsqql\"" Nov 02 17:33:31.938294 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.938126 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 17:33:31.938294 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.938264 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 17:33:31.938991 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.938716 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 17:33:31.938991 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.938760 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 17:33:31.939290 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.939272 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6p4gz\"" Nov 02 17:33:31.989548 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.989507 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 17:33:31.990045 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:31.990020 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 17:33:32.008295 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.008269 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8lqk\" (UniqueName: \"kubernetes.io/projected/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-api-access-k8lqk\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:32.008409 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.008316 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:32.008409 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.008372 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:32.008532 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.008405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:32.008532 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.008447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqn2p\" (UniqueName: \"kubernetes.io/projected/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-kube-api-access-pqn2p\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:32.008532 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.008471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/065a4c3d-712b-4b65-b18f-0428e7a93672-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:32.008532 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.008495 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:32.008742 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.008531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:32.008742 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.008556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:32.008742 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.008580 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/065a4c3d-712b-4b65-b18f-0428e7a93672-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:32.009424 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.009344 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:32.014252 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.014228 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:32.015968 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.015943 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:32.019084 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.019057 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqn2p\" (UniqueName: \"kubernetes.io/projected/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-kube-api-access-pqn2p\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:32.109021 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.108981 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.109179 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.109035 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-textfile\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.109179 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.109100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49hkf\" (UniqueName: \"kubernetes.io/projected/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-kube-api-access-49hkf\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.109179 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.109134 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:32.109179 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.109176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-root\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.109386 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.109213 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:32.109386 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.109238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-wtmp\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.109386 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.109265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:32.109386 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.109288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-tls\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.109386 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.109314 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.109386 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.109343 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/065a4c3d-712b-4b65-b18f-0428e7a93672-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:32.109386 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.109369 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/065a4c3d-712b-4b65-b18f-0428e7a93672-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:32.110976 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.109393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-metrics-client-ca\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.110976 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.109420 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-sys\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.110976 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.109446 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8lqk\" (UniqueName: \"kubernetes.io/projected/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-api-access-k8lqk\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:32.110976 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.110543 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/065a4c3d-712b-4b65-b18f-0428e7a93672-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:32.110976 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.110748 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/065a4c3d-712b-4b65-b18f-0428e7a93672-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:32.111218 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.111087 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:32.113737 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.113699 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:32.115455 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.115328 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:32.118132 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.118109 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8lqk\" (UniqueName: \"kubernetes.io/projected/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-api-access-k8lqk\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:32.194372 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.194339 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:32.210264 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.210165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-wtmp\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.210264 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.210209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-tls\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.210264 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.210236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.210466 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.210267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-metrics-client-ca\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.210466 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.210295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-sys\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.210466 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.210350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.210466 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.210382 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-textfile\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.210466 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.210443 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49hkf\" (UniqueName: \"kubernetes.io/projected/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-kube-api-access-49hkf\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.210729 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.210490 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-root\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.210729 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.210574 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-root\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.210729 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.210698 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-wtmp\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.212052 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.211110 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-sys\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.212052 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.211575 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-metrics-client-ca\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.212052 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.211674 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.212052 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.211770 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-textfile\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.213366 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.213324 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-tls\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.214272 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.214234 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.220495 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.220455 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49hkf\" (UniqueName: \"kubernetes.io/projected/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-kube-api-access-49hkf\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.230440 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.230391 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:32.235974 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.235823 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:32.488864 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.488684 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph"] Nov 02 17:33:32.500334 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:33:32.500301 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae7bace_4c4c_4f60_9863_5b2c29b54b76.slice/crio-6cdb434261cbe496fd69f7326173add7d06bf00c627e96c29e4e965338bbc464 WatchSource:0}: Error finding container 6cdb434261cbe496fd69f7326173add7d06bf00c627e96c29e4e965338bbc464: Status 404 returned error can't find the container with id 6cdb434261cbe496fd69f7326173add7d06bf00c627e96c29e4e965338bbc464 Nov 02 17:33:32.527429 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.527349 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477"] Nov 02 17:33:32.535600 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:33:32.535571 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065a4c3d_712b_4b65_b18f_0428e7a93672.slice/crio-d89c5699efb455fa88d5e2809af8e4806ee8c7a74c729b8bc09f276f9bea95c0 WatchSource:0}: Error finding container d89c5699efb455fa88d5e2809af8e4806ee8c7a74c729b8bc09f276f9bea95c0: Status 404 returned error can't find the container with id d89c5699efb455fa88d5e2809af8e4806ee8c7a74c729b8bc09f276f9bea95c0 Nov 02 17:33:32.888601 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.888532 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8dtcc" event={"ID":"0a0753ab-8636-4ce9-9b3e-d89516b73c1c","Type":"ContainerStarted","Data":"12e091b01a728b0f3d6d794d3f73abccbb037310e342e05e75ae7f4813fd484a"} Nov 02 17:33:32.891486 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.891447 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" event={"ID":"065a4c3d-712b-4b65-b18f-0428e7a93672","Type":"ContainerStarted","Data":"d89c5699efb455fa88d5e2809af8e4806ee8c7a74c729b8bc09f276f9bea95c0"} Nov 02 17:33:32.894528 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.894500 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" event={"ID":"7ae7bace-4c4c-4f60-9863-5b2c29b54b76","Type":"ContainerStarted","Data":"d6fe1f74510360238f2cf999b11f73f79ba47e85fe757345bc87c14112df2b21"} Nov 02 17:33:32.894663 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.894557 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" event={"ID":"7ae7bace-4c4c-4f60-9863-5b2c29b54b76","Type":"ContainerStarted","Data":"8ccd513fa1be21534cbf6ba8ca74327c1cbbc7d2609d16f2ce009f818c6b7b53"} Nov 02 17:33:32.894663 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:32.894584 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" event={"ID":"7ae7bace-4c4c-4f60-9863-5b2c29b54b76","Type":"ContainerStarted","Data":"6cdb434261cbe496fd69f7326173add7d06bf00c627e96c29e4e965338bbc464"} Nov 02 17:33:33.904424 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.904341 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rtxf5" event={"ID":"afbca029-7d59-4cfb-b08c-353e174b0355","Type":"ContainerStarted","Data":"f572468d01ff961aff704796807d81b6552c2f15dedc7dd2995d87be46ab553e"} Nov 02 17:33:33.904424 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.904404 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rtxf5" event={"ID":"afbca029-7d59-4cfb-b08c-353e174b0355","Type":"ContainerStarted","Data":"53821c5c5bf8991fa921a65fd0092b51faa7ada2ddcbd8e98936343cabd1f99d"} Nov 02 17:33:33.904951 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.904453 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:33.906267 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.906239 2579 generic.go:356] "Generic (PLEG): container finished" podID="0a0753ab-8636-4ce9-9b3e-d89516b73c1c" containerID="fb2560e050014c6afba8c9df47b9b554fbc9bce0e32d564539912e18e5d2a04e" exitCode=0 Nov 02 17:33:33.906381 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.906318 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8dtcc" event={"ID":"0a0753ab-8636-4ce9-9b3e-d89516b73c1c","Type":"ContainerDied","Data":"fb2560e050014c6afba8c9df47b9b554fbc9bce0e32d564539912e18e5d2a04e"} Nov 02 17:33:33.909752 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.909724 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" event={"ID":"7ae7bace-4c4c-4f60-9863-5b2c29b54b76","Type":"ContainerStarted","Data":"155c1f5b1db8d528af64eb607a1901a11468c6a52c492acac13eea719abef4ee"} Nov 02 17:33:33.912687 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.912664 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vrm5q" event={"ID":"9372be33-753d-4006-ba16-86fc44bd7c52","Type":"ContainerStarted","Data":"80a3cdcf71c506c51649c74782f92ad4c49c980ed86556807ff345315c0cbe14"} Nov 02 17:33:33.923099 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.922213 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-rtxf5" podStartSLOduration=1.7923745229999999 podStartE2EDuration="3.922196332s" totalImagesPullingTime="2.129821809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:33.921747507 +0000 UTC m=+49.135320965" watchObservedRunningTime="2026-11-02 17:33:33.922196332 +0000 UTC m=+49.135769766" Nov 02 17:33:33.967044 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.966902 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vrm5q" podStartSLOduration=1.5014572529999999 podStartE2EDuration="3.966888246s" totalImagesPullingTime="2.465430993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:33:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:33.94430371 +0000 UTC m=+49.157877142" watchObservedRunningTime="2026-11-02 17:33:33.966888246 +0000 UTC m=+49.180461678" Nov 02 17:33:33.967342 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.967312 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" podStartSLOduration=1.8154869630000001 podStartE2EDuration="2.967308532s" totalImagesPullingTime="1.151821569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:33.966546677 +0000 UTC m=+49.180120111" watchObservedRunningTime="2026-11-02 17:33:33.967308532 +0000 UTC m=+49.180881961" Nov 02 17:33:33.970880 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.970850 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6f98f548cc-srzfq"] Nov 02 17:33:33.974498 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.974469 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:33.978908 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.978887 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 17:33:33.979024 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.978955 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bjc4qab0pol8j\"" Nov 02 17:33:33.979104 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.978955 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 17:33:33.979327 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.978976 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 17:33:33.979381 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.979024 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 17:33:33.979422 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.979045 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 17:33:33.979468 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.979102 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fr655\"" Nov 02 17:33:33.990308 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:33.989023 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f98f548cc-srzfq"] Nov 02 17:33:34.025262 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.025237 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-tls\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.025365 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.025346 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnz65\" (UniqueName: \"kubernetes.io/projected/7ecf8c2f-987e-4441-a30d-2e75f3307aae-kube-api-access-tnz65\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.025428 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.025412 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.025490 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.025473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ecf8c2f-987e-4441-a30d-2e75f3307aae-tmp\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.025532 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.025510 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.025568 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.025535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ecf8c2f-987e-4441-a30d-2e75f3307aae-metrics-client-ca\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.025568 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.025558 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.025706 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.025581 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-grpc-tls\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.025706 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.025633 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.126758 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.126728 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnz65\" (UniqueName: \"kubernetes.io/projected/7ecf8c2f-987e-4441-a30d-2e75f3307aae-kube-api-access-tnz65\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.126864 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.126772 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.126864 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.126792 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ecf8c2f-987e-4441-a30d-2e75f3307aae-tmp\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.126864 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.126809 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.126864 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.126824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ecf8c2f-987e-4441-a30d-2e75f3307aae-metrics-client-ca\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.126864 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.126841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.126864 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.126856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-grpc-tls\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.127133 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.126877 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.127133 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.126900 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-tls\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.128630 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.128565 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ecf8c2f-987e-4441-a30d-2e75f3307aae-tmp\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.128630 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.128591 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ecf8c2f-987e-4441-a30d-2e75f3307aae-metrics-client-ca\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.130496 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.130368 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-grpc-tls\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.131265 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.131208 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.131348 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.131312 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.131821 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.131780 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.132767 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.132714 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.133726 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.133705 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-tls\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.138358 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.138313 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnz65\" (UniqueName: \"kubernetes.io/projected/7ecf8c2f-987e-4441-a30d-2e75f3307aae-kube-api-access-tnz65\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.297036 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.297004 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:34.581650 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.581587 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f98f548cc-srzfq"] Nov 02 17:33:34.584337 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:33:34.584311 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecf8c2f_987e_4441_a30d_2e75f3307aae.slice/crio-34af506efcaca75f8c72eb0547aa2968048bcaf45658b925a6acc5a2d52db5c2 WatchSource:0}: Error finding container 34af506efcaca75f8c72eb0547aa2968048bcaf45658b925a6acc5a2d52db5c2: Status 404 returned error can't find the container with id 34af506efcaca75f8c72eb0547aa2968048bcaf45658b925a6acc5a2d52db5c2 Nov 02 17:33:34.918685 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.918651 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8dtcc" event={"ID":"0a0753ab-8636-4ce9-9b3e-d89516b73c1c","Type":"ContainerStarted","Data":"ebf16562b83e733f2caf3488790d4cb32a13b16f975ea41dc3e4f62e275940f2"} Nov 02 17:33:34.919106 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.918711 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8dtcc" event={"ID":"0a0753ab-8636-4ce9-9b3e-d89516b73c1c","Type":"ContainerStarted","Data":"60c0b7370773deede9822f01e98abff85461bd4eb776a80d00d8d00cf8e88137"} Nov 02 17:33:34.920891 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.920861 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" event={"ID":"065a4c3d-712b-4b65-b18f-0428e7a93672","Type":"ContainerStarted","Data":"7711c726a7230707aebbc0980cfc7460827edb77e43a1a937ea29257d2c1a210"} Nov 02 17:33:34.921025 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.920921 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" event={"ID":"065a4c3d-712b-4b65-b18f-0428e7a93672","Type":"ContainerStarted","Data":"b49ce7037263bf819df3decef6166a22270c158b0d5499a4df27e9ddaa41951f"} Nov 02 17:33:34.921025 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.920947 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" event={"ID":"065a4c3d-712b-4b65-b18f-0428e7a93672","Type":"ContainerStarted","Data":"d3dea4ad3c468c6895f2df56c93286db6f235396d1fb38ee7fd3c7534d633b57"} Nov 02 17:33:34.922262 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.922053 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" event={"ID":"7ecf8c2f-987e-4441-a30d-2e75f3307aae","Type":"ContainerStarted","Data":"34af506efcaca75f8c72eb0547aa2968048bcaf45658b925a6acc5a2d52db5c2"} Nov 02 17:33:34.923415 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.923391 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nwbhl" event={"ID":"a47b32de-3a53-4b34-8874-87049c4d747b","Type":"ContainerStarted","Data":"b8b29c3892e5205154ca96e366e7fec04a194924c87a14ee01dd27bf8ad4dae2"} Nov 02 17:33:34.937767 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.937729 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8dtcc" podStartSLOduration=2.536089655 podStartE2EDuration="3.937718222s" totalImagesPullingTime="1.321633672s" totalInitContainerRuntime="79.994895ms" isStatefulPod=true podCreationTimestamp="2026-11-02 17:33:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:34.937276275 +0000 UTC m=+50.150849708" watchObservedRunningTime="2026-11-02 17:33:34.937718222 +0000 UTC m=+50.151291653" Nov 02 17:33:34.954081 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.954045 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nwbhl" podStartSLOduration=1.581239021 podStartE2EDuration="4.954034538s" totalImagesPullingTime="3.372795517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:34.954031477 +0000 UTC m=+50.167604909" watchObservedRunningTime="2026-11-02 17:33:34.954034538 +0000 UTC m=+50.167607970" Nov 02 17:33:34.975250 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:34.975211 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" podStartSLOduration=2.455133726 podStartE2EDuration="3.975200699s" totalImagesPullingTime="1.520066973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:34.97412476 +0000 UTC m=+50.187698188" watchObservedRunningTime="2026-11-02 17:33:34.975200699 +0000 UTC m=+50.188774130" Nov 02 17:33:36.400836 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.400797 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-dcf4d45cc-49kqs"] Nov 02 17:33:36.403311 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.403281 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.416879 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.416851 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-dcf4d45cc-49kqs"] Nov 02 17:33:36.418068 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.418046 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 17:33:36.418160 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.418113 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 17:33:36.443652 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.443599 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpp8l\" (UniqueName: \"kubernetes.io/projected/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-kube-api-access-dpp8l\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.443781 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.443670 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-secret-metrics-server-client-certs\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.443781 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.443762 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-audit-log\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.443880 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.443841 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-metrics-server-audit-profiles\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.443971 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.443937 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-client-ca-bundle\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.444079 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.443984 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-secret-metrics-server-tls\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.444079 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.444023 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.448975 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.448951 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f7cgovttces7q\"" Nov 02 17:33:36.449311 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.449293 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 17:33:36.450269 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.449676 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zrqks\"" Nov 02 17:33:36.450379 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.450355 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 17:33:36.544722 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.544688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpp8l\" (UniqueName: \"kubernetes.io/projected/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-kube-api-access-dpp8l\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.544722 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.544727 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-secret-metrics-server-client-certs\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.544944 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.544757 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-audit-log\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.545173 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.545151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-metrics-server-audit-profiles\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.545241 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.545155 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-audit-log\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.545241 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.545200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-client-ca-bundle\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.545326 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.545285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-secret-metrics-server-tls\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.545370 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.545325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.546033 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.546009 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.546141 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.546121 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-metrics-server-audit-profiles\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.547682 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.547660 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-secret-metrics-server-client-certs\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.547776 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.547698 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-client-ca-bundle\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.548402 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.548377 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-secret-metrics-server-tls\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.554392 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.554370 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpp8l\" (UniqueName: \"kubernetes.io/projected/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-kube-api-access-dpp8l\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.688529 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.688506 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4c892"] Nov 02 17:33:36.691158 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.691140 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4c892" Nov 02 17:33:36.695197 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.695181 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 17:33:36.695300 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.695283 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-kz9hl\"" Nov 02 17:33:36.699838 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.699818 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4c892"] Nov 02 17:33:36.717393 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.717372 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:36.746734 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.746701 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/43f970d6-97ec-4e75-820b-aceefb7c37f5-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4c892\" (UID: \"43f970d6-97ec-4e75-820b-aceefb7c37f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4c892" Nov 02 17:33:36.746825 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.746809 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwl2c\" (UniqueName: \"kubernetes.io/projected/43f970d6-97ec-4e75-820b-aceefb7c37f5-kube-api-access-wwl2c\") pod \"monitoring-plugin-77f4f76594-4c892\" (UID: \"43f970d6-97ec-4e75-820b-aceefb7c37f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4c892" Nov 02 17:33:36.847321 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.847298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwl2c\" (UniqueName: \"kubernetes.io/projected/43f970d6-97ec-4e75-820b-aceefb7c37f5-kube-api-access-wwl2c\") pod \"monitoring-plugin-77f4f76594-4c892\" (UID: \"43f970d6-97ec-4e75-820b-aceefb7c37f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4c892" Nov 02 17:33:36.847413 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.847377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/43f970d6-97ec-4e75-820b-aceefb7c37f5-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4c892\" (UID: \"43f970d6-97ec-4e75-820b-aceefb7c37f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4c892" Nov 02 17:33:36.849447 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.849428 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/43f970d6-97ec-4e75-820b-aceefb7c37f5-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4c892\" (UID: \"43f970d6-97ec-4e75-820b-aceefb7c37f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4c892" Nov 02 17:33:36.858290 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.858272 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwl2c\" (UniqueName: \"kubernetes.io/projected/43f970d6-97ec-4e75-820b-aceefb7c37f5-kube-api-access-wwl2c\") pod \"monitoring-plugin-77f4f76594-4c892\" (UID: \"43f970d6-97ec-4e75-820b-aceefb7c37f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4c892" Nov 02 17:33:36.931663 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.931635 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" event={"ID":"7ecf8c2f-987e-4441-a30d-2e75f3307aae","Type":"ContainerStarted","Data":"07dd32856654d8fcf45e88a927579f3a18014421bc806d76f351ef8011e491d6"} Nov 02 17:33:36.931807 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.931700 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" event={"ID":"7ecf8c2f-987e-4441-a30d-2e75f3307aae","Type":"ContainerStarted","Data":"25929c99fc0c741e243eb413d3759b3332965333e0d59908db6b2cdea80fe5e8"} Nov 02 17:33:36.931807 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.931731 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" event={"ID":"7ecf8c2f-987e-4441-a30d-2e75f3307aae","Type":"ContainerStarted","Data":"bdd77335322a49b5a45c14a2a771068eea5744539c5aeac7e482e72f0c1b9c04"} Nov 02 17:33:36.994144 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:36.994116 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-dcf4d45cc-49kqs"] Nov 02 17:33:36.997262 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:33:36.997234 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22db86a_ab8d_4a07_b3b2_6efdc24f8dc8.slice/crio-b707e4d118307bb1d4cef66e067f78387f8d64c93ed3a2590fa58e09d316f829 WatchSource:0}: Error finding container b707e4d118307bb1d4cef66e067f78387f8d64c93ed3a2590fa58e09d316f829: Status 404 returned error can't find the container with id b707e4d118307bb1d4cef66e067f78387f8d64c93ed3a2590fa58e09d316f829 Nov 02 17:33:37.044085 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:37.044063 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4c892" Nov 02 17:33:37.314648 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:37.314555 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4c892"] Nov 02 17:33:37.317305 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:33:37.317273 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f970d6_97ec_4e75_820b_aceefb7c37f5.slice/crio-01ac28ecaef8c0a2ff01290cd8d609c56c4916f9715a69cab0b8a8ec817cf75b WatchSource:0}: Error finding container 01ac28ecaef8c0a2ff01290cd8d609c56c4916f9715a69cab0b8a8ec817cf75b: Status 404 returned error can't find the container with id 01ac28ecaef8c0a2ff01290cd8d609c56c4916f9715a69cab0b8a8ec817cf75b Nov 02 17:33:37.939110 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:37.939074 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4c892" event={"ID":"43f970d6-97ec-4e75-820b-aceefb7c37f5","Type":"ContainerStarted","Data":"01ac28ecaef8c0a2ff01290cd8d609c56c4916f9715a69cab0b8a8ec817cf75b"} Nov 02 17:33:37.940435 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:37.940409 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" event={"ID":"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8","Type":"ContainerStarted","Data":"b707e4d118307bb1d4cef66e067f78387f8d64c93ed3a2590fa58e09d316f829"} Nov 02 17:33:37.943860 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:37.943352 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" event={"ID":"7ecf8c2f-987e-4441-a30d-2e75f3307aae","Type":"ContainerStarted","Data":"d2841e813d083fa0ad35d96e292591542a01b36ef39f1796b70ddd1d98aeb56d"} Nov 02 17:33:37.943860 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:37.943401 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" event={"ID":"7ecf8c2f-987e-4441-a30d-2e75f3307aae","Type":"ContainerStarted","Data":"97d4dc54f4d52332f36e9b7a11ba207fe66a4f74e31fa10c80ab6f3c2eae18ac"} Nov 02 17:33:37.943860 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:37.943426 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" event={"ID":"7ecf8c2f-987e-4441-a30d-2e75f3307aae","Type":"ContainerStarted","Data":"744051c95e7894f885426e17c0e08ff299483cd6324120df0d58848b316359aa"} Nov 02 17:33:37.943860 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:37.943732 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:37.973044 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:37.972986 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" podStartSLOduration=2.021952883 podStartE2EDuration="4.972972733s" totalImagesPullingTime="2.95101985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:37.970440677 +0000 UTC m=+53.184014108" watchObservedRunningTime="2026-11-02 17:33:37.972972733 +0000 UTC m=+53.186546193" Nov 02 17:33:38.179154 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.179116 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:33:38.182165 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.182143 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.186661 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.186637 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 17:33:38.186740 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.186692 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 17:33:38.186740 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.186712 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 17:33:38.187485 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.187459 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 17:33:38.187895 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.187610 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 17:33:38.187895 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.187864 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 17:33:38.188020 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.187893 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 17:33:38.188020 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.187893 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 17:33:38.188020 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.187908 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 17:33:38.188360 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.188346 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wrf7q\"" Nov 02 17:33:38.188511 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.188493 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cs83l4avv8s1v\"" Nov 02 17:33:38.188597 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.188538 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 17:33:38.188597 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.188592 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 17:33:38.188736 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.188703 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 17:33:38.188781 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.188751 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 17:33:38.193319 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.193297 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 17:33:38.201645 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.201064 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:33:38.260699 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.260669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.260863 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.260713 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.260863 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.260829 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.260973 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.260880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.260973 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.260938 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.261069 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.261018 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.261069 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.261055 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-web-config\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.261156 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.261087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.261156 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.261105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.261156 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.261119 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfvxh\" (UniqueName: \"kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-kube-api-access-dfvxh\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.261156 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.261135 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.261296 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.261162 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-config\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.261296 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.261187 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-config-out\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.261296 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.261210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.261296 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.261248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.261296 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.261277 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.261464 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.261303 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.261464 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.261327 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.261464 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.261357 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.261464 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.261385 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362309 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362488 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362322 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362488 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362488 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362488 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362412 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362488 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362488 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362468 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362879 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362526 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362879 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362879 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362879 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362649 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362879 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-web-config\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362879 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362702 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362879 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362729 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362879 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362757 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362879 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfvxh\" (UniqueName: \"kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-kube-api-access-dfvxh\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362879 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362809 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362879 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362836 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-config\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362879 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362859 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-config-out\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.362879 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.363451 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362905 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.363451 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.362922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.363451 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.363062 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.363451 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.363434 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.364144 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.363817 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.367031 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.366822 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.367695 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.367668 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-web-config\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.368188 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.367786 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.368188 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.368034 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.368349 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.368126 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.368349 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.368268 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.368349 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.368312 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.368491 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.368407 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.369284 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.369260 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.369799 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.369767 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.369930 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.369884 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-config-out\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.370310 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.370288 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.370888 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.370863 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-config\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.371825 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.371802 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.374867 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.374844 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfvxh\" (UniqueName: \"kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-kube-api-access-dfvxh\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.507895 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.507860 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:38.804368 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.804340 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:33:38.804794 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:33:38.804764 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ae9769_93f1_4962_b68c_22f5b4658484.slice/crio-1264654e7535055e82ca3e6ab8c24b022d08e7ededcad69edba5051e23b559fc WatchSource:0}: Error finding container 1264654e7535055e82ca3e6ab8c24b022d08e7ededcad69edba5051e23b559fc: Status 404 returned error can't find the container with id 1264654e7535055e82ca3e6ab8c24b022d08e7ededcad69edba5051e23b559fc Nov 02 17:33:38.947692 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.947658 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4c892" event={"ID":"43f970d6-97ec-4e75-820b-aceefb7c37f5","Type":"ContainerStarted","Data":"e1f8a74f4d073d24704fc9d3b656de10a38446e97a891cdb934e58cdaf1edde7"} Nov 02 17:33:38.948130 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.947844 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4c892" Nov 02 17:33:38.949329 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.949296 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" event={"ID":"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8","Type":"ContainerStarted","Data":"565da052ccf7726fb2530a7bddd8184581c93e36550dc032833261a94e960017"} Nov 02 17:33:38.950579 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.950559 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ae9769-93f1-4962-b68c-22f5b4658484","Type":"ContainerStarted","Data":"1264654e7535055e82ca3e6ab8c24b022d08e7ededcad69edba5051e23b559fc"} Nov 02 17:33:38.953464 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.953444 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4c892" Nov 02 17:33:38.965799 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.965735 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4c892" podStartSLOduration=1.574410158 podStartE2EDuration="2.965724846s" totalImagesPullingTime="1.391314688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:38.964372005 +0000 UTC m=+54.177945436" watchObservedRunningTime="2026-11-02 17:33:38.965724846 +0000 UTC m=+54.179298314" Nov 02 17:33:38.985774 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:38.985738 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" podStartSLOduration=1.595554199 podStartE2EDuration="2.985727631s" totalImagesPullingTime="1.390173432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:38.984842661 +0000 UTC m=+54.198416092" watchObservedRunningTime="2026-11-02 17:33:38.985727631 +0000 UTC m=+54.199301063" Nov 02 17:33:39.955152 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:39.955120 2579 generic.go:356] "Generic (PLEG): container finished" podID="54ae9769-93f1-4962-b68c-22f5b4658484" containerID="7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87" exitCode=0 Nov 02 17:33:39.955604 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:39.955209 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ae9769-93f1-4962-b68c-22f5b4658484","Type":"ContainerDied","Data":"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87"} Nov 02 17:33:42.981318 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:42.981134 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ae9769-93f1-4962-b68c-22f5b4658484","Type":"ContainerStarted","Data":"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128"} Nov 02 17:33:42.981318 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:42.981218 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ae9769-93f1-4962-b68c-22f5b4658484","Type":"ContainerStarted","Data":"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f"} Nov 02 17:33:42.981318 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:42.981260 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ae9769-93f1-4962-b68c-22f5b4658484","Type":"ContainerStarted","Data":"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc"} Nov 02 17:33:43.926102 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:43.926074 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:43.957339 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:43.957312 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:43.993380 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:43.993343 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ae9769-93f1-4962-b68c-22f5b4658484","Type":"ContainerStarted","Data":"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831"} Nov 02 17:33:43.993870 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:43.993429 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ae9769-93f1-4962-b68c-22f5b4658484","Type":"ContainerStarted","Data":"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6"} Nov 02 17:33:43.993870 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:43.993455 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ae9769-93f1-4962-b68c-22f5b4658484","Type":"ContainerStarted","Data":"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615"} Nov 02 17:33:44.022474 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:44.022225 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.239532998 podStartE2EDuration="6.022207233s" totalImagesPullingTime="3.767618583s" totalInitContainerRuntime="15.055652ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:44.02077836 +0000 UTC m=+59.234351792" watchObservedRunningTime="2026-11-02 17:33:44.022207233 +0000 UTC m=+59.235780667" Nov 02 17:33:45.932843 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:45.932814 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:48.428922 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:48.428885 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjnbh_c00d16e7-b92c-46c7-a68f-797274bbef93/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:48.508403 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:48.508373 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:48.579920 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:48.579894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjnbh_c00d16e7-b92c-46c7-a68f-797274bbef93/konnectivity-agent/0.log" limit=1024 Nov 02 17:33:50.075530 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:50.075498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tmr\" (UniqueName: \"kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr\") pod \"network-check-target-424d5\" (UID: \"9ce34815-aef2-487a-afd1-fe81200a423d\") " pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:50.075936 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:50.075553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:50.078717 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:50.078697 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 17:33:50.079579 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:50.079564 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 17:33:50.088852 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:50.088828 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:50.089776 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:50.089757 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 17:33:50.099726 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:50.099705 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9tmr\" (UniqueName: \"kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr\") pod \"network-check-target-424d5\" (UID: \"9ce34815-aef2-487a-afd1-fe81200a423d\") " pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:50.113228 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:50.113210 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dcw65\"" Nov 02 17:33:50.121438 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:50.121419 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:50.131610 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:50.131485 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-52q82\"" Nov 02 17:33:50.138826 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:50.138795 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:50.176869 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:50.176821 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs\") pod \"network-metrics-daemon-dfthz\" (UID: \"86286bdb-d69d-481a-b874-4ff3df42906f\") " pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:50.179981 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:50.179955 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 17:33:50.192160 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:50.192095 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs\") pod \"network-metrics-daemon-dfthz\" (UID: \"86286bdb-d69d-481a-b874-4ff3df42906f\") " pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:50.394938 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:50.394871 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f4xwb"] Nov 02 17:33:50.399060 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:33:50.398902 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29fb0ac_cd81_4d4c_bba7_42f9c0b686f7.slice/crio-2bd5334a4ba21443a748dd4a8a333e8e60d787cb084f227edb760c688af949b6 WatchSource:0}: Error finding container 2bd5334a4ba21443a748dd4a8a333e8e60d787cb084f227edb760c688af949b6: Status 404 returned error can't find the container with id 2bd5334a4ba21443a748dd4a8a333e8e60d787cb084f227edb760c688af949b6 Nov 02 17:33:50.420482 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:50.420459 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-424d5"] Nov 02 17:33:50.423433 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:33:50.423409 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce34815_aef2_487a_afd1_fe81200a423d.slice/crio-8d9fd7afb4bdd32b67dbf3631a5cbb23acaedef895044f9a3cb2b6d4479c5fb4 WatchSource:0}: Error finding container 8d9fd7afb4bdd32b67dbf3631a5cbb23acaedef895044f9a3cb2b6d4479c5fb4: Status 404 returned error can't find the container with id 8d9fd7afb4bdd32b67dbf3631a5cbb23acaedef895044f9a3cb2b6d4479c5fb4 Nov 02 17:33:50.426498 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:50.426482 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jmxfr\"" Nov 02 17:33:50.432589 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:50.432570 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:50.703136 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:50.703114 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dfthz"] Nov 02 17:33:50.705368 ip-10-0-141-169 kubenswrapper[2579]: W1102 17:33:50.705338 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86286bdb_d69d_481a_b874_4ff3df42906f.slice/crio-94a3a014356ddf1dafcfc55f48ee04dec8b93bad51b2b43874e23b17ad13992f WatchSource:0}: Error finding container 94a3a014356ddf1dafcfc55f48ee04dec8b93bad51b2b43874e23b17ad13992f: Status 404 returned error can't find the container with id 94a3a014356ddf1dafcfc55f48ee04dec8b93bad51b2b43874e23b17ad13992f Nov 02 17:33:51.022774 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:51.022687 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-424d5" event={"ID":"9ce34815-aef2-487a-afd1-fe81200a423d","Type":"ContainerStarted","Data":"8d9fd7afb4bdd32b67dbf3631a5cbb23acaedef895044f9a3cb2b6d4479c5fb4"} Nov 02 17:33:51.024259 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:51.024211 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dfthz" event={"ID":"86286bdb-d69d-481a-b874-4ff3df42906f","Type":"ContainerStarted","Data":"94a3a014356ddf1dafcfc55f48ee04dec8b93bad51b2b43874e23b17ad13992f"} Nov 02 17:33:51.025743 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:51.025715 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f4xwb" event={"ID":"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7","Type":"ContainerStarted","Data":"2bd5334a4ba21443a748dd4a8a333e8e60d787cb084f227edb760c688af949b6"} Nov 02 17:33:52.033780 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:52.033709 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dfthz" event={"ID":"86286bdb-d69d-481a-b874-4ff3df42906f","Type":"ContainerStarted","Data":"cd3baa8c0298188b54300ce9006e2786afac78e56005582d378cf7b1fa9e5119"} Nov 02 17:33:53.040985 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:53.040104 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dfthz" event={"ID":"86286bdb-d69d-481a-b874-4ff3df42906f","Type":"ContainerStarted","Data":"e80567761227a45ed7989a4a36b465fe5fbf23e6288b18549ea6a2b5baa87c6a"} Nov 02 17:33:53.067173 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:53.067116 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dfthz" podStartSLOduration=65.938356838 podStartE2EDuration="1m7.0670984s" totalImagesPullingTime="1.128741562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:53.066739823 +0000 UTC m=+68.280313256" watchObservedRunningTime="2026-11-02 17:33:53.0670984 +0000 UTC m=+68.280671834" Nov 02 17:33:54.046647 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:54.046597 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-424d5" event={"ID":"9ce34815-aef2-487a-afd1-fe81200a423d","Type":"ContainerStarted","Data":"21990bae1b6ad92349dd6036c8bbe71b83d3a65d20f8f07481336eeb0dcc64d1"} Nov 02 17:33:54.047138 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:54.046922 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:54.064457 ip-10-0-141-169 kubenswrapper[2579]: I1102 17:33:54.064403 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-424d5" podStartSLOduration=64.84584254 podStartE2EDuration="1m8.064386008s" totalImagesPullingTime="3.218543468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:54.063346186 +0000 UTC m=+69.276919622" watchObservedRunningTime="2026-11-02 17:33:54.064386008 +0000 UTC m=+69.277959442" Nov 02 17:33:55.252118 ip-10-0-141-169 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 17:33:55.265038 ip-10-0-141-169 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 17:33:55.265251 ip-10-0-141-169 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 17:33:55.265444 ip-10-0-141-169 systemd[1]: kubelet.service: Consumed 4.138s CPU time. Nov 02 17:33:55.277152 ip-10-0-141-169 systemd[1]: Starting Kubernetes Kubelet... Nov 02 17:33:55.338034 ip-10-0-141-169 kubenswrapper[5601]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:33:55.338034 ip-10-0-141-169 kubenswrapper[5601]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 17:33:55.338034 ip-10-0-141-169 kubenswrapper[5601]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:33:55.338034 ip-10-0-141-169 kubenswrapper[5601]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:33:55.340377 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340285 5601 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:33:55.340377 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340372 5601 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:33:55.340377 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340378 5601 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:33:55.340377 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340382 5601 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:33:55.340798 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340386 5601 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:33:55.340798 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340391 5601 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:33:55.340798 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340394 5601 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:33:55.340798 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340398 5601 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:33:55.340798 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340402 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:33:55.340798 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340406 5601 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:33:55.340798 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340410 5601 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:33:55.340798 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340413 5601 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:33:55.340798 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340419 5601 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:33:55.340798 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340426 5601 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:33:55.340798 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340432 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:33:55.340798 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340437 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:33:55.340798 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340441 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:33:55.340798 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340446 5601 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:33:55.340798 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340451 5601 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:33:55.340798 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340455 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:33:55.340798 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340459 5601 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:33:55.340798 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340465 5601 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:33:55.340798 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340471 5601 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:33:55.341609 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340475 5601 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:33:55.341609 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340480 5601 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:33:55.341609 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340484 5601 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:33:55.341609 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340488 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:33:55.341609 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340492 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:33:55.341609 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340496 5601 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:33:55.341609 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340500 5601 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:33:55.341609 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340505 5601 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:33:55.341609 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340509 5601 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:33:55.341609 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340513 5601 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:33:55.341609 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340518 5601 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:33:55.341609 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340523 5601 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:33:55.341609 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340528 5601 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:33:55.341609 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340532 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:33:55.341609 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340537 5601 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:33:55.341609 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340541 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:33:55.341609 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340545 5601 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:33:55.341609 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340549 5601 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:33:55.341609 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340554 5601 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340559 5601 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340564 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340568 5601 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340573 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340577 5601 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340581 5601 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340585 5601 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340589 5601 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340593 5601 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340598 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340603 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340607 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340628 5601 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340633 5601 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340637 5601 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340641 5601 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340646 5601 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340650 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340654 5601 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:33:55.342467 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340658 5601 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340662 5601 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340666 5601 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340671 5601 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340677 5601 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340682 5601 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340686 5601 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340690 5601 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340694 5601 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340698 5601 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340703 5601 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340707 5601 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340711 5601 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340716 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340720 5601 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340724 5601 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340729 5601 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340734 5601 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340738 5601 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340742 5601 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:33:55.343348 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340746 5601 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:33:55.344275 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340751 5601 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:33:55.344275 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340755 5601 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:33:55.344275 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340760 5601 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:33:55.344275 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340764 5601 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:33:55.344275 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340772 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:33:55.344275 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340777 5601 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:33:55.344275 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340782 5601 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:33:55.344275 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340786 5601 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:33:55.344275 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340790 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:33:55.344275 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340794 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:33:55.344275 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340798 5601 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:33:55.344275 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340802 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:33:55.344275 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340806 5601 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:33:55.344275 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340810 5601 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:33:55.344275 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340814 5601 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:33:55.344275 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340819 5601 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:33:55.344275 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340824 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:33:55.344275 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340829 5601 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:33:55.344275 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340833 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:33:55.345140 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340837 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:33:55.345140 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340842 5601 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:33:55.345140 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340846 5601 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:33:55.345140 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340850 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:33:55.345140 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.340855 5601 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:33:55.345140 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342163 5601 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:33:55.345140 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342174 5601 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:33:55.345140 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342179 5601 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:33:55.345140 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342187 5601 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:33:55.345140 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342193 5601 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:33:55.345140 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342199 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:33:55.345140 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342203 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:33:55.345140 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342208 5601 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:33:55.345140 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342213 5601 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:33:55.345140 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342217 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:33:55.345140 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342222 5601 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:33:55.345140 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342226 5601 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:33:55.345140 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342231 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:33:55.345140 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342236 5601 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342241 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342245 5601 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342250 5601 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342255 5601 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342259 5601 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342264 5601 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342268 5601 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342272 5601 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342277 5601 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342281 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342287 5601 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342292 5601 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342297 5601 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342302 5601 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342306 5601 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342311 5601 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342315 5601 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342320 5601 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342324 5601 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:33:55.346014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342328 5601 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:33:55.346899 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342333 5601 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:33:55.346899 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342337 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:33:55.346899 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342341 5601 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:33:55.346899 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342346 5601 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:33:55.346899 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342350 5601 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:33:55.346899 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342355 5601 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:33:55.346899 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342359 5601 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:33:55.346899 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342364 5601 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:33:55.346899 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342368 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:33:55.346899 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342372 5601 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:33:55.346899 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342376 5601 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:33:55.346899 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342381 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:33:55.346899 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342385 5601 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:33:55.346899 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342390 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:33:55.346899 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342394 5601 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:33:55.346899 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342398 5601 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:33:55.346899 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342403 5601 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:33:55.346899 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342407 5601 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:33:55.346899 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342412 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342416 5601 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342421 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342425 5601 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342430 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342434 5601 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342439 5601 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342443 5601 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342448 5601 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342452 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342456 5601 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342461 5601 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342465 5601 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342469 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342473 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342477 5601 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342481 5601 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342485 5601 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342489 5601 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342493 5601 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:33:55.347400 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342498 5601 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342502 5601 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342507 5601 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342511 5601 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342516 5601 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342520 5601 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342524 5601 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342529 5601 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342533 5601 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342537 5601 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342542 5601 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342546 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342551 5601 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342556 5601 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342560 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342565 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342569 5601 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342573 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342577 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342582 5601 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:33:55.347922 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342586 5601 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:33:55.348471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342590 5601 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:33:55.348471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342598 5601 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:33:55.348471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342604 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:33:55.348471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342609 5601 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:33:55.348471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342631 5601 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:33:55.348471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342637 5601 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:33:55.348471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342642 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:33:55.348471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342646 5601 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:33:55.348471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342650 5601 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:33:55.348471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342655 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:33:55.348471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342666 5601 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:33:55.348471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342670 5601 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:33:55.348471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.342675 5601 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:33:55.348471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343481 5601 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:33:55.348471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343490 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:33:55.348471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343494 5601 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:33:55.348471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343499 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:33:55.348471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343503 5601 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:33:55.348471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343508 5601 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343512 5601 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343518 5601 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343522 5601 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343529 5601 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343534 5601 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343539 5601 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343543 5601 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343548 5601 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343552 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343556 5601 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343561 5601 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343566 5601 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343571 5601 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343575 5601 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343580 5601 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343610 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343636 5601 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343642 5601 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343647 5601 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:33:55.349019 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343651 5601 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343656 5601 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343660 5601 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343664 5601 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343671 5601 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343676 5601 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343680 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343685 5601 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343689 5601 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343694 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343698 5601 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343703 5601 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343707 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343711 5601 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343716 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343721 5601 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343725 5601 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343730 5601 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343734 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343738 5601 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:33:55.349689 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343742 5601 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:33:55.350399 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343747 5601 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:33:55.350399 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343751 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:33:55.350399 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343755 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:33:55.350399 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343760 5601 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:33:55.350399 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343765 5601 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:33:55.350399 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343770 5601 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:33:55.350399 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343774 5601 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:33:55.350399 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343779 5601 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:33:55.350399 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343783 5601 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:33:55.350399 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343787 5601 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:33:55.350399 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343791 5601 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:33:55.350399 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343795 5601 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:33:55.350399 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343800 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:33:55.350399 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343804 5601 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:33:55.350399 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343809 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:33:55.350399 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343814 5601 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:33:55.350399 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343819 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:33:55.350399 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343823 5601 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:33:55.350399 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343828 5601 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343832 5601 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343836 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343841 5601 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343846 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343872 5601 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343878 5601 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343883 5601 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343888 5601 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343893 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343901 5601 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343907 5601 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343912 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343917 5601 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343921 5601 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343926 5601 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343931 5601 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343935 5601 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343940 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343945 5601 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:33:55.350881 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343950 5601 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:33:55.351361 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343954 5601 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:33:55.351361 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343958 5601 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:33:55.351361 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343962 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:33:55.351361 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343966 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:33:55.351361 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343971 5601 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:33:55.351361 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343976 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:33:55.351361 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343980 5601 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:33:55.351361 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343986 5601 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:33:55.351361 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343992 5601 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:33:55.351361 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.343996 5601 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:33:55.351361 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.344000 5601 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:33:55.351361 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.344005 5601 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:33:55.351361 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.344009 5601 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:33:55.351361 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.344013 5601 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:33:55.351361 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.344017 5601 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:33:55.351361 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.344022 5601 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:33:55.351361 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.344026 5601 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:33:55.351361 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.344031 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:33:55.351361 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.344035 5601 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.344039 5601 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.344043 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344185 5601 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344202 5601 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344212 5601 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344219 5601 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344226 5601 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344231 5601 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344243 5601 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344250 5601 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344255 5601 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344260 5601 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344267 5601 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344272 5601 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344277 5601 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344282 5601 flags.go:64] FLAG: --cgroup-root="" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344286 5601 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344291 5601 flags.go:64] FLAG: --client-ca-file="" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344295 5601 flags.go:64] FLAG: --cloud-provider="external" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344301 5601 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344309 5601 flags.go:64] FLAG: --cluster-domain="" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344316 5601 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 17:33:55.351852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344322 5601 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344328 5601 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344334 5601 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344340 5601 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344345 5601 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344349 5601 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344355 5601 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344360 5601 flags.go:64] FLAG: --contention-profiling="false" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344365 5601 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344370 5601 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344375 5601 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344380 5601 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344387 5601 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344392 5601 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344397 5601 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344402 5601 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344411 5601 flags.go:64] FLAG: --enable-server="true" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344416 5601 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344422 5601 flags.go:64] FLAG: --event-burst="100" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344428 5601 flags.go:64] FLAG: --event-qps="50" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344432 5601 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344437 5601 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344442 5601 flags.go:64] FLAG: --eviction-hard="" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344450 5601 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 17:33:55.352415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344455 5601 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344460 5601 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344465 5601 flags.go:64] FLAG: --eviction-soft="" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344470 5601 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344475 5601 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344480 5601 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344485 5601 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344492 5601 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344497 5601 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344502 5601 flags.go:64] FLAG: --feature-gates="" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344509 5601 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344514 5601 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344519 5601 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344524 5601 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344529 5601 flags.go:64] FLAG: --healthz-port="10248" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344534 5601 flags.go:64] FLAG: --help="false" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344539 5601 flags.go:64] FLAG: --hostname-override="ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344545 5601 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344550 5601 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344554 5601 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344560 5601 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344566 5601 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344571 5601 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 17:33:55.353021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344575 5601 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344583 5601 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344587 5601 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344592 5601 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344597 5601 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344602 5601 flags.go:64] FLAG: --kube-reserved="" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344607 5601 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344629 5601 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344635 5601 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344641 5601 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344646 5601 flags.go:64] FLAG: --lock-file="" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344651 5601 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344655 5601 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344661 5601 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344670 5601 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344674 5601 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344681 5601 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344686 5601 flags.go:64] FLAG: --logging-format="text" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344691 5601 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344697 5601 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344701 5601 flags.go:64] FLAG: --manifest-url="" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344706 5601 flags.go:64] FLAG: --manifest-url-header="" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344713 5601 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344718 5601 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344725 5601 flags.go:64] FLAG: --max-pods="110" Nov 02 17:33:55.353578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344730 5601 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344735 5601 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344740 5601 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344744 5601 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344749 5601 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344754 5601 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344758 5601 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344770 5601 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344777 5601 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344782 5601 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344787 5601 flags.go:64] FLAG: --pod-cidr="" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344792 5601 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344796 5601 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344802 5601 flags.go:64] FLAG: --pods-per-core="0" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344806 5601 flags.go:64] FLAG: --port="10250" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344812 5601 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344816 5601 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-03667256e140917f5" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344823 5601 flags.go:64] FLAG: --qos-reserved="" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344828 5601 flags.go:64] FLAG: --read-only-port="10255" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344833 5601 flags.go:64] FLAG: --register-node="true" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344838 5601 flags.go:64] FLAG: --register-with-taints="" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344844 5601 flags.go:64] FLAG: --registry-burst="10" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344849 5601 flags.go:64] FLAG: --registry-qps="5" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344856 5601 flags.go:64] FLAG: --reserved-cpus="" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344861 5601 flags.go:64] FLAG: --reserved-memory="" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344867 5601 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 17:33:55.354192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344872 5601 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344877 5601 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344882 5601 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344887 5601 flags.go:64] FLAG: --runonce="false" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344891 5601 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344896 5601 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344901 5601 flags.go:64] FLAG: --seccomp-default="false" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344906 5601 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344911 5601 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344916 5601 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344921 5601 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344926 5601 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344931 5601 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344936 5601 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344944 5601 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344948 5601 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344954 5601 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344959 5601 flags.go:64] FLAG: --system-cgroups="" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344963 5601 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344972 5601 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344976 5601 flags.go:64] FLAG: --tls-cert-file="" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344981 5601 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344989 5601 flags.go:64] FLAG: --tls-min-version="" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344993 5601 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.344999 5601 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 17:33:55.354849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.345004 5601 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.345009 5601 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.345014 5601 flags.go:64] FLAG: --v="2" Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.345021 5601 flags.go:64] FLAG: --version="false" Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.345030 5601 flags.go:64] FLAG: --vmodule="" Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.345036 5601 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.345042 5601 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345258 5601 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345267 5601 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345272 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345276 5601 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345281 5601 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345286 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345291 5601 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345295 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345300 5601 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345304 5601 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345309 5601 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345314 5601 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345318 5601 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345323 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:33:55.355554 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345329 5601 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:33:55.356165 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345334 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:33:55.356165 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345338 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:33:55.356165 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345342 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:33:55.356165 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345346 5601 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:33:55.356165 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345351 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:33:55.356165 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345355 5601 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:33:55.356165 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345360 5601 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:33:55.356165 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345364 5601 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:33:55.356165 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345368 5601 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:33:55.356165 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345373 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:33:55.356165 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345378 5601 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:33:55.356165 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345382 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:33:55.356165 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345387 5601 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:33:55.356165 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345391 5601 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:33:55.356165 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345396 5601 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:33:55.356165 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345400 5601 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:33:55.356165 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345405 5601 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:33:55.356165 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345409 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:33:55.356165 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345413 5601 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:33:55.356660 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345420 5601 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:33:55.356660 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345425 5601 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:33:55.356660 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345429 5601 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:33:55.356660 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345435 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:33:55.356660 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345440 5601 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:33:55.356660 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345444 5601 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:33:55.356660 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345449 5601 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:33:55.356660 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345453 5601 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:33:55.356660 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345457 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:33:55.356660 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345462 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:33:55.356660 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345468 5601 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:33:55.356660 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345475 5601 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:33:55.356660 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345483 5601 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:33:55.356660 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345488 5601 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:33:55.356660 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345493 5601 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:33:55.356660 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345498 5601 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:33:55.356660 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345503 5601 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:33:55.356660 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345508 5601 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:33:55.356660 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345512 5601 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345516 5601 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345521 5601 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345525 5601 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345529 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345534 5601 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345540 5601 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345545 5601 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345549 5601 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345553 5601 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345558 5601 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345562 5601 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345567 5601 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345572 5601 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345576 5601 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345580 5601 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345585 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345589 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345594 5601 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345598 5601 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:33:55.357173 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345603 5601 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345607 5601 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345627 5601 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345632 5601 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345636 5601 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345640 5601 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345647 5601 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345651 5601 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345656 5601 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345660 5601 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345664 5601 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345668 5601 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345673 5601 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345677 5601 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345681 5601 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345686 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345690 5601 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345694 5601 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345699 5601 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345704 5601 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:33:55.357666 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345708 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:33:55.358244 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345713 5601 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:33:55.358244 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345717 5601 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:33:55.358244 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345722 5601 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:33:55.358244 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345726 5601 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:33:55.358244 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345730 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:33:55.358244 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345735 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:33:55.358244 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345739 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:33:55.358244 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345744 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:33:55.358244 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345748 5601 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:33:55.358244 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345752 5601 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:33:55.358244 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345756 5601 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:33:55.358244 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345761 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:33:55.358244 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345765 5601 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:33:55.358244 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.345808 5601 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:33:55.358244 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345866 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:33:55.358244 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345874 5601 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:33:55.358705 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345879 5601 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:33:55.358705 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345884 5601 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:33:55.358705 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345890 5601 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:33:55.358705 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345894 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:33:55.358705 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345899 5601 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:33:55.358705 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345903 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:33:55.358705 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345907 5601 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:33:55.358705 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345912 5601 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:33:55.358705 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345916 5601 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:33:55.358705 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345920 5601 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:33:55.358705 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345924 5601 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:33:55.358705 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345929 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:33:55.358705 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345934 5601 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:33:55.358705 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345940 5601 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:33:55.358705 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345946 5601 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:33:55.358705 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345951 5601 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:33:55.358705 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345956 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:33:55.358705 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345960 5601 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:33:55.358705 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345964 5601 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:33:55.359293 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345969 5601 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:33:55.359293 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345973 5601 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:33:55.359293 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345978 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:33:55.359293 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345982 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:33:55.359293 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345989 5601 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:33:55.359293 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.345995 5601 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:33:55.359293 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346001 5601 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:33:55.359293 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346005 5601 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:33:55.359293 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346010 5601 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:33:55.359293 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346014 5601 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:33:55.359293 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346018 5601 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:33:55.359293 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346022 5601 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:33:55.359293 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346026 5601 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:33:55.359293 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346030 5601 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:33:55.359293 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346036 5601 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:33:55.359293 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346042 5601 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:33:55.359293 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346046 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:33:55.359293 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346051 5601 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:33:55.359293 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346056 5601 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346061 5601 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346065 5601 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346069 5601 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346076 5601 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346080 5601 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346085 5601 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346089 5601 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346093 5601 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346098 5601 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346102 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346108 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346112 5601 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346117 5601 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346121 5601 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346126 5601 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346130 5601 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346135 5601 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346139 5601 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346144 5601 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346148 5601 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:33:55.359913 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346152 5601 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346156 5601 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346160 5601 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346165 5601 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346169 5601 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346173 5601 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346178 5601 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346184 5601 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346191 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346195 5601 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346199 5601 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346204 5601 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346208 5601 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346212 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346217 5601 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346221 5601 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346226 5601 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346230 5601 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346235 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346239 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:33:55.360452 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346243 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:33:55.360964 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346247 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:33:55.360964 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346252 5601 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:33:55.360964 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346258 5601 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:33:55.360964 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346262 5601 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:33:55.360964 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346267 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:33:55.360964 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346271 5601 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:33:55.360964 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346275 5601 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:33:55.360964 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346280 5601 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:33:55.360964 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346284 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:33:55.360964 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346288 5601 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:33:55.360964 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346293 5601 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:33:55.360964 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346297 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:33:55.360964 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346301 5601 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:33:55.360964 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346306 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:33:55.360964 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346310 5601 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:33:55.360964 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346316 5601 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:33:55.360964 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346321 5601 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:33:55.360964 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346325 5601 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:33:55.360964 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346331 5601 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:33:55.361449 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346337 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:33:55.361449 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346341 5601 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:33:55.361449 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346346 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:33:55.361449 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346350 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:33:55.361449 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346354 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:33:55.361449 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.346359 5601 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:33:55.361449 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.351148 5601 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 17:33:55.361449 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.351165 5601 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 17:33:55.361449 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.351183 5601 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 17:33:55.361449 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.351189 5601 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 17:33:55.361449 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351249 5601 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:33:55.361449 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351254 5601 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:33:55.361449 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351258 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:33:55.361449 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351262 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:33:55.361449 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351265 5601 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:33:55.361449 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351268 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:33:55.361449 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351271 5601 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:33:55.361449 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351274 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:33:55.361449 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351277 5601 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:33:55.361950 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351279 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:33:55.361950 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351282 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:33:55.361950 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351285 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:33:55.361950 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351288 5601 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:33:55.361950 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351290 5601 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:33:55.361950 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351293 5601 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:33:55.361950 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351301 5601 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:33:55.361950 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351304 5601 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:33:55.361950 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351307 5601 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:33:55.361950 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351309 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:33:55.361950 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351312 5601 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:33:55.361950 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351315 5601 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:33:55.361950 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351318 5601 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:33:55.361950 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351321 5601 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:33:55.361950 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351323 5601 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:33:55.361950 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351326 5601 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:33:55.361950 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351329 5601 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:33:55.361950 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351331 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:33:55.361950 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351335 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:33:55.362525 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351338 5601 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:33:55.362525 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351341 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:33:55.362525 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351344 5601 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:33:55.362525 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351347 5601 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:33:55.362525 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351350 5601 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:33:55.362525 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351353 5601 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:33:55.362525 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351355 5601 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:33:55.362525 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351358 5601 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:33:55.362525 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351360 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:33:55.362525 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351363 5601 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:33:55.362525 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351366 5601 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:33:55.362525 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351371 5601 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:33:55.362525 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351375 5601 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:33:55.362525 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351378 5601 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:33:55.362525 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351381 5601 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:33:55.362525 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351384 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:33:55.362525 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351387 5601 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:33:55.362525 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351389 5601 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:33:55.362525 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351397 5601 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351400 5601 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351403 5601 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351406 5601 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351408 5601 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351411 5601 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351413 5601 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351416 5601 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351418 5601 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351421 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351424 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351427 5601 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351430 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351433 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351436 5601 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351439 5601 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351442 5601 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351445 5601 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351447 5601 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351450 5601 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:33:55.363038 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351452 5601 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:33:55.363591 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351455 5601 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:33:55.363591 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351459 5601 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:33:55.363591 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351462 5601 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:33:55.363591 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351465 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:33:55.363591 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351468 5601 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:33:55.363591 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351471 5601 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:33:55.363591 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351474 5601 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:33:55.363591 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351477 5601 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:33:55.363591 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351479 5601 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:33:55.363591 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351482 5601 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:33:55.363591 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351484 5601 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:33:55.363591 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351487 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:33:55.363591 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351494 5601 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:33:55.363591 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351497 5601 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:33:55.363591 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351500 5601 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:33:55.363591 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351503 5601 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:33:55.363591 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351505 5601 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:33:55.363591 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351508 5601 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:33:55.363591 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351511 5601 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351513 5601 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351517 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351519 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351522 5601 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351524 5601 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351527 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351530 5601 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351533 5601 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351536 5601 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351538 5601 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351541 5601 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351543 5601 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351546 5601 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351548 5601 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351551 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351553 5601 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351556 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351558 5601 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351561 5601 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:33:55.364305 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351563 5601 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:33:55.364809 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.351590 5601 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:33:55.364809 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351747 5601 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:33:55.364809 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351754 5601 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:33:55.364809 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351757 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:33:55.364809 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351760 5601 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:33:55.364809 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351763 5601 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:33:55.364809 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351766 5601 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:33:55.364809 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351769 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:33:55.364809 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351771 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:33:55.364809 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351774 5601 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:33:55.364809 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351777 5601 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:33:55.364809 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351779 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:33:55.364809 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351782 5601 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:33:55.364809 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351784 5601 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:33:55.364809 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351787 5601 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:33:55.364809 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351790 5601 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351792 5601 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351795 5601 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351797 5601 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351800 5601 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351803 5601 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351805 5601 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351808 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351810 5601 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351813 5601 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351816 5601 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351818 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351820 5601 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351823 5601 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351826 5601 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351828 5601 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351831 5601 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351833 5601 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351836 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351838 5601 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:33:55.365191 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351841 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:33:55.365682 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351844 5601 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:33:55.365682 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351848 5601 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:33:55.365682 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351852 5601 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:33:55.365682 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351856 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:33:55.365682 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351859 5601 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:33:55.365682 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351862 5601 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:33:55.365682 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351864 5601 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:33:55.365682 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351867 5601 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:33:55.365682 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351870 5601 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:33:55.365682 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351873 5601 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:33:55.365682 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351876 5601 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:33:55.365682 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351879 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:33:55.365682 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351882 5601 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:33:55.365682 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351884 5601 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:33:55.365682 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351887 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:33:55.365682 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351889 5601 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:33:55.365682 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351892 5601 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:33:55.365682 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351895 5601 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:33:55.365682 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351897 5601 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351900 5601 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351902 5601 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351905 5601 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351907 5601 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351910 5601 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351912 5601 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351915 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351917 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351920 5601 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351923 5601 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351925 5601 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351928 5601 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351931 5601 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351933 5601 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351936 5601 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351939 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351942 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351945 5601 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351948 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:33:55.366430 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351950 5601 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351953 5601 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351955 5601 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351958 5601 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351961 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351964 5601 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351967 5601 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351969 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351972 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351974 5601 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351977 5601 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351979 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351982 5601 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351984 5601 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351987 5601 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351990 5601 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351992 5601 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351995 5601 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.351997 5601 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352000 5601 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:33:55.367237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352003 5601 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:33:55.367800 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352005 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:33:55.367800 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352008 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:33:55.367800 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352011 5601 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:33:55.367800 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352013 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:33:55.367800 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352016 5601 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:33:55.367800 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352018 5601 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:33:55.367800 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352021 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:33:55.367800 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352023 5601 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:33:55.367800 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352026 5601 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:33:55.367800 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352028 5601 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:33:55.367800 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352031 5601 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:33:55.367800 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352034 5601 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:33:55.367800 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.352056 5601 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:33:55.367800 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352089 5601 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:33:55.367800 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352094 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:33:55.367800 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352097 5601 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:33:55.368237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352100 5601 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:33:55.368237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352103 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:33:55.368237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352106 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:33:55.368237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352109 5601 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:33:55.368237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352111 5601 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:33:55.368237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352122 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:33:55.368237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352125 5601 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:33:55.368237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352128 5601 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:33:55.368237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352131 5601 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:33:55.368237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352133 5601 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:33:55.368237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352136 5601 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:33:55.368237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352139 5601 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:33:55.368237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352141 5601 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:33:55.368237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352147 5601 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:33:55.368237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352149 5601 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:33:55.368237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352152 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:33:55.368237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352155 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:33:55.368237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352158 5601 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:33:55.368237 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352161 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352164 5601 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352167 5601 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352169 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352172 5601 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352175 5601 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352177 5601 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352180 5601 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352182 5601 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352187 5601 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352191 5601 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352194 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352197 5601 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352200 5601 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352203 5601 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352206 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352208 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352211 5601 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352214 5601 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352216 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:33:55.368713 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352219 5601 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352221 5601 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352224 5601 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352227 5601 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352229 5601 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352232 5601 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352234 5601 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352240 5601 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352243 5601 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352246 5601 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352249 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352252 5601 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352254 5601 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352257 5601 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352260 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352263 5601 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352265 5601 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352268 5601 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352270 5601 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352273 5601 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:33:55.369197 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352276 5601 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:33:55.369692 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352278 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:33:55.369692 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352282 5601 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:33:55.369692 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352284 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:33:55.369692 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352287 5601 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:33:55.369692 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352291 5601 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:33:55.369692 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352295 5601 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:33:55.369692 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352298 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:33:55.369692 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352301 5601 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:33:55.369692 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352304 5601 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:33:55.369692 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352307 5601 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:33:55.369692 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352309 5601 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:33:55.369692 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352312 5601 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:33:55.369692 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352315 5601 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:33:55.369692 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352317 5601 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:33:55.369692 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352320 5601 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:33:55.369692 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352322 5601 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:33:55.369692 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352325 5601 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:33:55.369692 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352328 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:33:55.369692 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352331 5601 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352334 5601 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352337 5601 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352340 5601 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352342 5601 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352345 5601 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352347 5601 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352350 5601 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352353 5601 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352356 5601 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352358 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352361 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352363 5601 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352366 5601 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352368 5601 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352371 5601 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352374 5601 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352377 5601 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352381 5601 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352384 5601 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:33:55.370223 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352386 5601 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:33:55.370744 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352389 5601 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:33:55.370744 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352392 5601 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:33:55.370744 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352394 5601 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:33:55.370744 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352398 5601 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:33:55.370744 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:33:55.352400 5601 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:33:55.370744 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.352457 5601 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 17:33:55.370744 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.353391 5601 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 17:33:55.370744 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.353441 5601 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 17:33:55.370744 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.353628 5601 server.go:1014] "Starting client certificate rotation" Nov 02 17:33:55.370744 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.353761 5601 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:33:55.370744 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.353816 5601 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:27:46 +0000 UTC" deadline="2028-05-11 04:27:18.566081558 +0000 UTC" Nov 02 17:33:55.370744 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.353844 5601 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13330h53m23.212240527s" Nov 02 17:33:55.370744 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.354063 5601 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:33:55.370744 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.355068 5601 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:33:55.370744 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.356274 5601 log.go:25] "Validated CRI v1 runtime API" Nov 02 17:33:55.370744 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.358197 5601 log.go:25] "Validated CRI v1 image API" Nov 02 17:33:55.371131 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.358937 5601 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 17:33:55.371131 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.361021 5601 fs.go:135] Filesystem UUIDs: map[5ff19fdd-8fb7-4c34-b8b9-284ff3d1a8ec:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 921a8f18-a4a4-4bde-9044-b365c11be3b1:/dev/nvme0n1p3] Nov 02 17:33:55.371217 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.361044 5601 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01ac28ecaef8c0a2ff01290cd8d609c56c4916f9715a69cab0b8a8ec817cf75b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01ac28ecaef8c0a2ff01290cd8d609c56c4916f9715a69cab0b8a8ec817cf75b/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06a75e22443636b5d6f9595c1a91faf891a1814859883ba9d367b81fab5ae7f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06a75e22443636b5d6f9595c1a91faf891a1814859883ba9d367b81fab5ae7f3/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/077f6bade89779397e191fbf68451bda0f46f291e1b2a9f81bfee8688a9d1c07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/077f6bade89779397e191fbf68451bda0f46f291e1b2a9f81bfee8688a9d1c07/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1264654e7535055e82ca3e6ab8c24b022d08e7ededcad69edba5051e23b559fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1264654e7535055e82ca3e6ab8c24b022d08e7ededcad69edba5051e23b559fc/userdata/shm major:0 minor:383 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12e091b01a728b0f3d6d794d3f73abccbb037310e342e05e75ae7f4813fd484a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12e091b01a728b0f3d6d794d3f73abccbb037310e342e05e75ae7f4813fd484a/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2771f15ce74dbd7bbb1485be9914a1064a4f50bdf3e0552ffb65daab89654c24/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2771f15ce74dbd7bbb1485be9914a1064a4f50bdf3e0552ffb65daab89654c24/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2bd5334a4ba21443a748dd4a8a333e8e60d787cb084f227edb760c688af949b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2bd5334a4ba21443a748dd4a8a333e8e60d787cb084f227edb760c688af949b6/userdata/shm major:0 minor:430 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2de52cc070fc04494fe6bae9e958fde642017d687e5bc27b5230393cefc5b6ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2de52cc070fc04494fe6bae9e958fde642017d687e5bc27b5230393cefc5b6ca/userdata/shm major:0 minor:65 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34af506efcaca75f8c72eb0547aa2968048bcaf45658b925a6acc5a2d52db5c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34af506efcaca75f8c72eb0547aa2968048bcaf45658b925a6acc5a2d52db5c2/userdata/shm major:0 minor:317 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6cdb434261cbe496fd69f7326173add7d06bf00c627e96c29e4e965338bbc464/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6cdb434261cbe496fd69f7326173add7d06bf00c627e96c29e4e965338bbc464/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73e7fa334d83301b9af102efa79afce3f99e00e42080633b168bbcd8a22c9fee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73e7fa334d83301b9af102efa79afce3f99e00e42080633b168bbcd8a22c9fee/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c2265955010341f47ae43e2242c473921e2d11089d45d940bfbc87218662811/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c2265955010341f47ae43e2242c473921e2d11089d45d940bfbc87218662811/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c97a3b904cb02796a1e696cc3a9a28c0baa6b091b3366784a6b7c8c039a7488/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c97a3b904cb02796a1e696cc3a9a28c0baa6b091b3366784a6b7c8c039a7488/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d9fd7afb4bdd32b67dbf3631a5cbb23acaedef895044f9a3cb2b6d4479c5fb4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d9fd7afb4bdd32b67dbf3631a5cbb23acaedef895044f9a3cb2b6d4479c5fb4/userdata/shm major:0 minor:432 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94a3a014356ddf1dafcfc55f48ee04dec8b93bad51b2b43874e23b17ad13992f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94a3a014356ddf1dafcfc55f48ee04dec8b93bad51b2b43874e23b17ad13992f/userdata/shm major:0 minor:439 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b707e4d118307bb1d4cef66e067f78387f8d64c93ed3a2590fa58e09d316f829/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b707e4d118307bb1d4cef66e067f78387f8d64c93ed3a2590fa58e09d316f829/userdata/shm major:0 minor:331 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bac27cb3c407150e65218780c36da0ea0a465950911c6e1ef696403ab454bc4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bac27cb3c407150e65218780c36da0ea0a465950911c6e1ef696403ab454bc4d/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1cacef195d5be9ba06c4120bc166fa099b012a5b5e9eadce24a287821c1064f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1cacef195d5be9ba06c4120bc166fa099b012a5b5e9eadce24a287821c1064f/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d8567d7af3f898368be2a2d0cbb7633ab84ea10911fa1f6c561d05bb3d8d371b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d8567d7af3f898368be2a2d0cbb7633ab84ea10911fa1f6c561d05bb3d8d371b/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d89c5699efb455fa88d5e2809af8e4806ee8c7a74c729b8bc09f276f9bea95c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d89c5699efb455fa88d5e2809af8e4806ee8c7a74c729b8bc09f276f9bea95c0/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dcc69eae5c7774f8b56c3f671e21766dce7bf983a0337b36318d9b06cc76d13d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dcc69eae5c7774f8b56c3f671e21766dce7bf983a0337b36318d9b06cc76d13d/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec0c7d25ba197d2d809535834a87240820ff73dde8bd8883fd36a6279c5cd644/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec0c7d25ba197d2d809535834a87240820ff73dde8bd8883fd36a6279c5cd644/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef1fd04818d9c3135fea78c802e9766f59d1009c48cf7395e53284f02f7ddded/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef1fd04818d9c3135fea78c802e9766f59d1009c48cf7395e53284f02f7ddded/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/065a4c3d-712b-4b65-b18f-0428e7a93672/volumes/kubernetes.io~projected/kube-api-access-k8lqk:{mountpoint:/var/lib/kubelet/pods/065a4c3d-712b-4b65-b18f-0428e7a93672/volumes/kubernetes.io~projected/kube-api-access-k8lqk major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/065a4c3d-712b-4b65-b18f-0428e7a93672/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/065a4c3d-712b-4b65-b18f-0428e7a93672/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/065a4c3d-712b-4b65-b18f-0428e7a93672/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/065a4c3d-712b-4b65-b18f-0428e7a93672/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a0753ab-8636-4ce9-9b3e-d89516b73c1c/volumes/kubernetes.io~projected/kube-api-access-49hkf:{mountpoint:/var/lib/kubelet/pods/0a0753ab-8636-4ce9-9b3e-d89516b73c1c/volumes/kubernetes.io~projected/kube-api-access-49hkf major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a0753ab-8636-4ce9-9b3e-d89516b73c1c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0a0753ab-8636-4ce9-9b3e-d89516b73c1c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a0753ab-8636-4ce9-9b3e-d89516b73c1c/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/0a0753ab-8636-4ce9-9b3e-d89516b73c1c/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11bb6b24-b4e9-4764-a40a-e1daa2c613c2/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/11bb6b24-b4e9-4764-a40a-e1daa2c613c2/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11bb6b24-b4e9-4764-a40a-e1daa2c613c2/volumes/kubernetes.io~projected/kube-api-access-cnt46:{mountpoint:/var/lib/kubelet/pods/11bb6b24-b4e9-4764-a40a-e1daa2c613c2/volumes/kubernetes.io~projected/kube-api-access-cnt46 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11bb6b24-b4e9-4764-a40a-e1daa2c613c2/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/11bb6b24-b4e9-4764-a40a-e1daa2c613c2/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e/volumes/kubernetes.io~projected/kube-api-access-t97rb:{mountpoint:/var/lib/kubelet/pods/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e/volumes/kubernetes.io~projected/kube-api-access-t97rb major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e35ffc8-ef10-4532-af91-38841c3f4581/volumes/kubernetes.io~projected/kube-api-access-mlzk8:{mountpoint:/var/lib/kubelet/pods/3e35ffc8-ef10-4532-af91-38841c3f4581/volumes/kubernetes.io~projected/kube-api-access-mlzk8 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40e6a790-de24-49f1-b507-ed74d1446868/volumes/kubernetes.io~projected/kube-api-access-ptckf:{mountpoint:/var/lib/kubelet/pods/40e6a790-de24-49f1-b507-ed74d1446868/volumes/kubernetes.io~projected/kube-api-access-ptckf major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43f970d6-97ec-4e75-820b-aceefb7c37f5/volumes/kubernetes.io~projected/kube-api-access-wwl2c:{mountpoint:/var/lib/kubelet/pods/43f970d6-97ec-4e75-820b-aceefb7c37f5/volumes/kubernetes.io~projected/kube-api-access-wwl2c major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43f970d6-97ec-4e75-820b-aceefb7c37f5/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/43f970d6-97ec-4e75-820b-aceefb7c37f5/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44174ea2-6704-423f-a176-c3abe64be85e/volumes/kubernetes.io~projected/kube-api-access-95kp2:{mountpoint:/var/lib/kubelet/pods/44174ea2-6704-423f-a176-c3abe64be85e/volumes/kubernetes.io~projected/kube-api-access-95kp2 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volume-subpaths/web-config/config-reloader/2 major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volume-subpaths/web-config/prometheus/15 major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~empty-dir/config-out major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~projected/kube-api-access-dfvxh:{mountpoint:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~projected/kube-api-access-dfvxh major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~projected/tls-assets major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/config major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/web-config major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ae7bace-4c4c-4f60-9863-5b2c29b54b76/volumes/kubernetes.io~projected/kube-api-access-pqn2p:{mountpoint:/var/lib/kubelet/pods/7ae7bace-4c4c-4f60-9863-5b2c29b54b76/volumes/kubernetes.io~projected/kube-api-access-pqn2p major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ae7bace-4c4c-4f60-9863-5b2c29b54b76/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/7ae7bace-4c4c-4f60-9863-5b2c29b54b76/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ae7bace-4c4c-4f60-9863-5b2c29b54b76/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/7ae7bace-4c4c-4f60-9863-5b2c29b54b76/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~projected/kube-api-access-tnz65:{mountpoint:/var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~projected/kube-api-access-tnz65 major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86286bdb-d69d-481a-b874-4ff3df42906f/volumes/kubernetes.io~projected/kube-api-access-lqxvs:{mountpoint:/var/lib/kubelet/pods/86286bdb-d69d-481a-b874-4ff3df42906f/volumes/kubernetes.io~projected/kube-api-access-lqxvs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86286bdb-d69d-481a-b874-4ff3df42906f/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/86286bdb-d69d-481a-b874-4ff3df42906f/volumes/kubernetes.io~secret/metrics-certs major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d225fb4-f477-4971-bce1-8aaa24a79be8/volumes/kubernetes.io~projected/kube-api-access-kwnst:{mountpoint:/var/lib/kubelet/pods/8d225fb4-f477-4971-bce1-8aaa24a79be8/volumes/kubernetes.io~projected/kube-api-access-kwnst major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9372be33-753d-4006-ba16-86fc44bd7c52/volumes/kubernetes.io~projected/kube-api-access-rpg54:{mountpoint:/var/lib/kubelet/pods/9372be33-753d-4006-ba16-86fc44bd7c52/volumes/kubernetes.io~projected/kube-api-access-rpg54 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9372be33-753d-4006-ba16-86fc44bd7c52/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/9372be33-753d-4006-ba16-86fc44bd7c52/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ce34815-aef2-487a-afd1-fe81200a423d/volumes/kubernetes.io~projected/kube-api-access-m9tmr:{mountpoint:/var/lib/kubelet/pods/9ce34815-aef2-487a-afd1-fe81200a423d/volumes/kubernetes.io~projected/kube-api-access-m9tmr major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a47b32de-3a53-4b34-8874-87049c4d747b/volumes/kubernetes.io~projected/kube-api-access-7vjcv:{mountpoint:/var/lib/kubelet/pods/a47b32de-3a53-4b34-8874-87049c4d747b/volumes/kubernetes.io~projected/kube-api-access-7vjcv major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a47b32de-3a53-4b34-8874-87049c4d747b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a47b32de-3a53-4b34-8874-87049c4d747b/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc/volumes/kubernetes.io~projected/kube-api-access-4662z:{mountpoint:/var/lib/kubelet/pods/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc/volumes/kubernetes.io~projected/kube-api-access-4662z major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afbca029-7d59-4cfb-b08c-353e174b0355/volumes/kubernetes.io~projected/kube-api-access-hdccp:{mountpoint:/var/lib/kubelet/pods/afbca029-7d59-4cfb-b08c-353e174b0355/volumes/kubernetes.io~projected/kube-api-access-hdccp major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afbca029-7d59-4cfb-b08c-353e174b0355/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/afbca029-7d59-4cfb-b08c-353e174b0355/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c00d16e7-b92c-46c7-a68f-797274bbef93/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c00d16e7-b92c-46c7-a68f-797274bbef93/volumes/kubernetes.io~secret/agent-certs major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8/volumes/kubernetes.io~projected/kube-api-access-dpp8l:{mountpoint:/var/lib/kubelet/pods/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8/volumes/kubernetes.io~projected/kube-api-access-dpp8l major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9300fb6-6926-4429-8797-603dfb361818/volumes/kubernetes.io~projected/kube-api-access-m4lln:{mountpoint:/var/lib/kubelet/pods/d9300fb6-6926-4429-8797-603dfb361818/volumes/kubernetes.io~projected/kube-api-access-m4lln major:0 minor:54 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/e9ab0f61a335521ad31d81a142fd1b37839d95cebbbfd8361bbe41157af51b02/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/68c081ae97f1c9dd6122fa556c753e2c3bbb34d8db4fcab5566a0ed2c74cd8d8/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c90052f0717b427ca5eef4755181d7376cd45c773e8e20b1a146df7c4b85848c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/dc8dea07b6f4afa32f363bdeda28aed08a0f4a7270c7b7a68c0df40f000517e7/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/5e28fdd46cfa4b190c3fda23d8729808820bc882432d38aacfaa1daead3a7782/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/073764518d9923ad34a9e69b4b030579cbdadff5a6074c83c8f3b9f5b3b1dbd8/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5f46bf51c39ce28206024b5e85a62c59e0d98d07e59009efd19cab140363ad4b/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f4e53e92b1b1a39f252eae9f3e9df9b5a41f5a9e38c6ed4a965d8ed4f6fa6b8b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/ee89d873834f88df352522de946ac1260d971dc9c00ff94f86b010d027b5fda2/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/ca794b04e27ad26f5b6995396ee91b1d45916b21c80e1cffac30fbe66cf06663/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/7d5cfa962b7f13a5d6746893c55083ae25a8123303a5598b4b420e2ff97422d4/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/f4b0197e14538a3eea2f58b234da8ae79753a102ee92404cec2c1569217bc610/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e9def86dcb8674489e32d62a64eaa3850a5b022d5a18e3cb3a9741f11a346a61/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/0b2843a97d28b5e688c262111c0c4d5ec20391554dbe283e311d6805f1919a7d/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/db95b47fe0111021f4cfcb2f16cbb2fe92ae1994d83b1b8ee25557856efb3513/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/2c5146288efcb9c7a102ff82104dc0046599ff6bfd7f6ed4ef3c5aeded2c4438/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/1c755d7881af85b72734d182c26021e4bf71ffc716e8328bdf4de0755aaf8a36/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a86e7b6010e86bf67a85e52e2591a11a9e5e16b29ea039711fc3b7b3950c5e5d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/003528a6de44e9ccf7b35ad0d98e382143f17431b10884c31f982008bd8ad0c8/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/ef44d5fcefeffd5986f23cc2e1a5a75ea7bf9473dad6f2d3ded1fca62823b7d9/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f2c459dc4afca3fc694c2a73c85d7c3bb51349c2dbdd6d359cd680b06fc3ac31/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/7ae8a0dac3d3c718be0b92d91689c81a87a8ebb50534544641223d9099d97507/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/836b2754286aaad530a4ec45146a585f8efe3ef100625095bdeebe65f17a4776/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/9e3cd88dcbeccb52eb9776e94b50173c4b68c8889b95bebaf99b3ae416135212/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/4d61b3cf42392364750c5f02b28d113c6c502458b00d64392609ff4c5a7eb8ab/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/eab8e98557caa5aaa96b8bcaa96579efd6480ec36784b51a6981e659cdfcf4d7/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/f534777dc334190c74e00aecfa4810c584daa957896002ae68563981854aafbf/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/ac717488b9354ed5170de73ff419c981b2a65db68517ae375cd6041eb56aa21d/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/e852a3962daa09cc66622b1cd74804f4a2442cc459ad3d238a9b42838e438bf0/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/a52ec768255865cfdf9d90591ffb1b99b474cf0cbc8d0ba3212779de7b154cd4/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/67e0b9a022293423e34beb1590851363416bf2ccf9e27e8d7448c7ef2d74046d/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/e6ff4867bbecbb327b90a91dc8d3042b8c165fe40a1d3dbc8f8450d7c3d23f98/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/f553a48adabd2a9a289608094efd470ee1956dc55572909b7c0ba27a5e647b5d/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/d20e0f8c23806c4f9473e4c6b9ce87a59f714805ecf3991b50af50ae3a4d8d2b/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/4c2e30b63767879771fa8dd85e4b174e4f105b1fbab065c9ef2016d2647559f0/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/3157ac33e63d076ab17eae38c9034ba96d861cc4feb390b10518b9dba31e827f/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/3fe363ed550240d1a598d46350c665122b0ce5672205148761848c83e4bec5ff/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/a7c42865a5b8a273f443c69e19c50ed2be2432e88ad38b4f2b911ea7be7c010e/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/c4669fb075153c927c0d3bcd963c8faccdefa3bebe7efe9b81c511327ef44e3b/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/e5a90f1f92cd2a5c18c0c5afb04dff10234131a22479d8a2ef0af8fe71b0296e/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/15f25d1469fe5425d1b957effc568b17b6e561ad83031b5e83e11679d7c2c09a/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/3a64a93c38aca4c81e7a59808048db2d36793d8b18ece12f2229634664898957/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/82b6a181479fbea2d891414f189393b8c57aa385ec48a9e8d7f877d444406570/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/ad046b1515cf560ac7b19f3aa8e20ec8a97c284cea6bacdf42300f16117e7243/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/a5b8faf25eefe048f37c6906cac44a6b34b9986e06fc11785183c1370fc8782c/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/e95d9b7c934f114277e14c57d71d5a15fa080075e858d726a6a69b34d3e20cb7/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/61993c99c8d85c834cad2ab5f61269eb4bcf58d8a12861e8ff853c8731af60cd/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/f86c66ec2d28b4da2db843d9fdbd73d490b4553b1b1676fa09b78bb0de8ba5aa/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/e7aa0da41994521bf92dd5e5a20b3f5a911425dc67cffac572a8cb0c6d73f455/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/3b6093ea4b9500b8e1683d56934b5eb3f8ae9ab536f4631944b4965a223b8c80/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/8523a4aecde45f097d69959f6ea4fde41cde0ffe1f92f0797301532a59589967/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/e14eb39a2bdf17b40e75aee947ac665ec717e3efe96d53bb33483ac45653aae4/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/5479cce2014af4b60c48e67af95ae1b9dc5fe5398b35c9e2a87d17209fa50129/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/bdd5718ee393070e8855afbd61c9f1f36dfb7146e701be1be3b0df3d794deb60/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/fe78e7a0c976353fa62e7c6cb05f299c60844966ae6c7da051824b4be58ad2e1/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/5e09a3674caf0f0adcb813814b2187559621aa4a4889a524f8ba9119582cafdf/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/46eba3b5ee0156625cc1677bd2ceb5834309f21523fafc896df01fb8629d0344/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/f6153c6da87030cf54ceb63df5e1768c670af06ffb74f4a9715536bdad0f343d/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/79c4aa29ecee3607c7af62460bbbb6842bbd05e26e1ff37a3610990e9d67cf31/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-435:{mountpoint:/var/lib/containers/storage/overlay/70875fe435625852a668dd30da424c6a6226beea189ebadd13bfccda9ac91075/merged major:0 minor:435 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/7f44198ef039b120cb947ee70ee26226cf4ad705554325f97c81815d323fef06/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/0af6d3c56db722b456915194eb23d96eabd027e7aa092e3d735ad6a188e71e13/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/773bc01e0d4ccdaa462e80b0fe4cd6b0d4bfdd26ce6abe253d049c77ad553c11/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-445:{mountpoint:/var/lib/containers/storage/overlay/aa0e8363099acc1514afd6a4349aa7eb593a40a7c92397fb56e8b71f7d7659a2/merged major:0 minor:445 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/01b7252fedb376c89db2dd1e5b18b5be784043ead32185eb612dc27b5525f7e3/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/0d3a8190bb1a72fda0b84147d55d8957c251966a458f9b523a65dcf12114cb18/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/343e48c52cae4ff26ee199ea7fccf3c2a5cd432ba3d26d7157ed7278fc240fdb/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/a90bd76312080c1891072f38fc2624491966ae6ddbc37b49bdbb7a01dab2a800/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/f4f2fa2929db615d4a561ac04b6f8777eb5b5341b1d483dafdd74e38a793e8ca/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/bba626423d0bc3da24a5c7e66252a8c376e150340c9e082ba764a959215f04a0/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/d9aaa5c05d072191513a48b336ca485fd1d8357d2c95db692076a994e30bae54/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/0e50e1899cef3aa6145caf9de5b44c0f1957eca7388f833d39538e68f8402144/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/6e9232fe07614020c2a668d7f5bd1ba750db27a411fb2ed01782a7f15a4e5e96/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/fcdc9c4ceadd2b552c4a94c0e85825eda247d1db2f7da27506bb65c3ea95acec/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/919dc96b74b1f9a5497b3c36985ca4abca64374f17c2cb242d5790200f6b3220/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/8ec55270d703c40298c4952d63abeebd8430d9d7763662067984ee6ab2dbc6d1/merged major:0 minor:97 fsType:overlay blockSize:0}] Nov 02 17:33:55.371477 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.370673 5601 manager.go:217] Machine: {Timestamp:2026-11-02 17:33:55.370182267 +0000 UTC m=+0.061236270 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3113082 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22aa1c762d8ea629a72adce91f0c8b SystemUUID:ec22aa1c-762d-8ea6-29a7-2adce91f0c8b BootID:7a7ca67d-c634-4e85-a310-6a573ae23c48 Filesystems:[{Device:/run/containers/storage/overlay-containers/8d9fd7afb4bdd32b67dbf3631a5cbb23acaedef895044f9a3cb2b6d4479c5fb4/userdata/shm DeviceMajor:0 DeviceMinor:432 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/11bb6b24-b4e9-4764-a40a-e1daa2c613c2/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34af506efcaca75f8c72eb0547aa2968048bcaf45658b925a6acc5a2d52db5c2/userdata/shm DeviceMajor:0 DeviceMinor:317 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:425 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/9372be33-753d-4006-ba16-86fc44bd7c52/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/065a4c3d-712b-4b65-b18f-0428e7a93672/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:231 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:324 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:377 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-445 DeviceMajor:0 DeviceMinor:445 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:300 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43f970d6-97ec-4e75-820b-aceefb7c37f5/volumes/kubernetes.io~projected/kube-api-access-wwl2c DeviceMajor:0 DeviceMinor:348 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c2265955010341f47ae43e2242c473921e2d11089d45d940bfbc87218662811/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:302 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8/volumes/kubernetes.io~projected/kube-api-access-dpp8l DeviceMajor:0 DeviceMinor:327 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:373 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:366 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a47b32de-3a53-4b34-8874-87049c4d747b/volumes/kubernetes.io~projected/kube-api-access-7vjcv DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11bb6b24-b4e9-4764-a40a-e1daa2c613c2/volumes/kubernetes.io~projected/kube-api-access-cnt46 DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:376 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc/volumes/kubernetes.io~projected/kube-api-access-4662z DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:301 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:371 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1264654e7535055e82ca3e6ab8c24b022d08e7ededcad69edba5051e23b559fc/userdata/shm DeviceMajor:0 DeviceMinor:383 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ae7bace-4c4c-4f60-9863-5b2c29b54b76/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06a75e22443636b5d6f9595c1a91faf891a1814859883ba9d367b81fab5ae7f3/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bac27cb3c407150e65218780c36da0ea0a465950911c6e1ef696403ab454bc4d/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:366 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/c00d16e7-b92c-46c7-a68f-797274bbef93/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ae7bace-4c4c-4f60-9863-5b2c29b54b76/volumes/kubernetes.io~projected/kube-api-access-pqn2p DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/065a4c3d-712b-4b65-b18f-0428e7a93672/volumes/kubernetes.io~projected/kube-api-access-k8lqk DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d89c5699efb455fa88d5e2809af8e4806ee8c7a74c729b8bc09f276f9bea95c0/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40e6a790-de24-49f1-b507-ed74d1446868/volumes/kubernetes.io~projected/kube-api-access-ptckf DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec0c7d25ba197d2d809535834a87240820ff73dde8bd8883fd36a6279c5cd644/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7ae7bace-4c4c-4f60-9863-5b2c29b54b76/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01ac28ecaef8c0a2ff01290cd8d609c56c4916f9715a69cab0b8a8ec817cf75b/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:370 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/86286bdb-d69d-481a-b874-4ff3df42906f/volumes/kubernetes.io~projected/kube-api-access-lqxvs DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-435 DeviceMajor:0 DeviceMinor:435 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94a3a014356ddf1dafcfc55f48ee04dec8b93bad51b2b43874e23b17ad13992f/userdata/shm DeviceMajor:0 DeviceMinor:439 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a47b32de-3a53-4b34-8874-87049c4d747b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/afbca029-7d59-4cfb-b08c-353e174b0355/volumes/kubernetes.io~projected/kube-api-access-hdccp DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~projected/kube-api-access-tnz65 DeviceMajor:0 DeviceMinor:305 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a0753ab-8636-4ce9-9b3e-d89516b73c1c/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:236 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0a0753ab-8636-4ce9-9b3e-d89516b73c1c/volumes/kubernetes.io~projected/kube-api-access-49hkf DeviceMajor:0 DeviceMinor:238 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:325 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef1fd04818d9c3135fea78c802e9766f59d1009c48cf7395e53284f02f7ddded/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:298 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9ce34815-aef2-487a-afd1-fe81200a423d/volumes/kubernetes.io~projected/kube-api-access-m9tmr DeviceMajor:0 DeviceMinor:429 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86286bdb-d69d-481a-b874-4ff3df42906f/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:434 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9372be33-753d-4006-ba16-86fc44bd7c52/volumes/kubernetes.io~projected/kube-api-access-rpg54 DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11bb6b24-b4e9-4764-a40a-e1daa2c613c2/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73e7fa334d83301b9af102efa79afce3f99e00e42080633b168bbcd8a22c9fee/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/065a4c3d-712b-4b65-b18f-0428e7a93672/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:232 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:374 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b707e4d118307bb1d4cef66e067f78387f8d64c93ed3a2590fa58e09d316f829/userdata/shm DeviceMajor:0 DeviceMinor:331 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:372 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e/volumes/kubernetes.io~projected/kube-api-access-t97rb DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dcc69eae5c7774f8b56c3f671e21766dce7bf983a0337b36318d9b06cc76d13d/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/afbca029-7d59-4cfb-b08c-353e174b0355/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:375 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2bd5334a4ba21443a748dd4a8a333e8e60d787cb084f227edb760c688af949b6/userdata/shm DeviceMajor:0 DeviceMinor:430 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2771f15ce74dbd7bbb1485be9914a1064a4f50bdf3e0552ffb65daab89654c24/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0a0753ab-8636-4ce9-9b3e-d89516b73c1c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:237 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:326 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:366 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:366 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7ecf8c2f-987e-4441-a30d-2e75f3307aae/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:299 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e35ffc8-ef10-4532-af91-38841c3f4581/volumes/kubernetes.io~projected/kube-api-access-mlzk8 DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1cacef195d5be9ba06c4120bc166fa099b012a5b5e9eadce24a287821c1064f/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/44174ea2-6704-423f-a176-c3abe64be85e/volumes/kubernetes.io~projected/kube-api-access-95kp2 DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c97a3b904cb02796a1e696cc3a9a28c0baa6b091b3366784a6b7c8c039a7488/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2de52cc070fc04494fe6bae9e958fde642017d687e5bc27b5230393cefc5b6ca/userdata/shm DeviceMajor:0 DeviceMinor:65 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/8d225fb4-f477-4971-bce1-8aaa24a79be8/volumes/kubernetes.io~projected/kube-api-access-kwnst DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6cdb434261cbe496fd69f7326173add7d06bf00c627e96c29e4e965338bbc464/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:379 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~projected/kube-api-access-dfvxh DeviceMajor:0 DeviceMinor:380 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d9300fb6-6926-4429-8797-603dfb361818/volumes/kubernetes.io~projected/kube-api-access-m4lln DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d8567d7af3f898368be2a2d0cbb7633ab84ea10911fa1f6c561d05bb3d8d371b/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:378 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/077f6bade89779397e191fbf68451bda0f46f291e1b2a9f81bfee8688a9d1c07/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12e091b01a728b0f3d6d794d3f73abccbb037310e342e05e75ae7f4813fd484a/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/43f970d6-97ec-4e75-820b-aceefb7c37f5/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:344 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:01ac28ecaef8c0a MacAddress:a6:5d:2b:ca:7e:4e Speed:10000 Mtu:8901} {Name:077f6bade897793 MacAddress:82:64:b0:d3:e1:30 Speed:10000 Mtu:8901} {Name:1264654e7535055 MacAddress:52:dd:7c:ee:29:a0 Speed:10000 Mtu:8901} {Name:2771f15ce74dbd7 MacAddress:92:e2:27:c7:27:11 Speed:10000 Mtu:8901} {Name:2bd5334a4ba2144 MacAddress:f2:c0:ea:ce:5d:19 Speed:10000 Mtu:8901} {Name:34af506efcaca75 MacAddress:92:c4:ad:bf:df:e0 Speed:10000 Mtu:8901} {Name:6cdb434261cbe49 MacAddress:32:47:c1:a9:73:69 Speed:10000 Mtu:8901} {Name:73e7fa334d83301 MacAddress:e2:f6:84:97:1a:c4 Speed:10000 Mtu:8901} {Name:8d9fd7afb4bdd32 MacAddress:72:80:d5:27:c5:22 Speed:10000 Mtu:8901} {Name:94a3a014356ddf1 MacAddress:e6:7a:9b:75:3e:ff Speed:10000 Mtu:8901} {Name:b707e4d118307bb MacAddress:82:11:f9:a9:bb:1d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d5:5b:69:53 Speed:0 Mtu:9001} {Name:br-int MacAddress:86:c5:b4:34:25:03 Speed:0 Mtu:8901} {Name:d89c5699efb455f MacAddress:f6:9e:f1:f3:5d:6d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d5:5b:69:53 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:42:f4:05:62:a0:a0 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:72:54:15:96:ca:4e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 17:33:55.371477 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.371055 5601 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 17:33:55.371477 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.371099 5601 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 17:33:55.371477 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.371380 5601 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 17:33:55.371938 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.371405 5601 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-169.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 17:33:55.371938 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.371530 5601 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 17:33:55.371938 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.371538 5601 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 17:33:55.371938 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.371548 5601 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:33:55.371938 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.371560 5601 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:33:55.371938 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.371673 5601 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 17:33:55.371938 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.371839 5601 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 17:33:55.372208 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.371944 5601 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 17:33:55.372208 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.371977 5601 kubelet.go:493] "Attempting to sync node with API server" Nov 02 17:33:55.372208 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.371990 5601 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 17:33:55.372208 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.372000 5601 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 17:33:55.372208 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.372008 5601 kubelet.go:405] "Adding apiserver pod source" Nov 02 17:33:55.372208 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.372016 5601 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 17:33:55.372709 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.372694 5601 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 17:33:55.372709 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.372706 5601 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:33:55.373201 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.373177 5601 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 17:33:55.376108 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.376093 5601 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 17:33:55.376108 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.376107 5601 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:33:55.376225 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.376215 5601 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 17:33:55.376414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.376404 5601 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 17:33:55.376445 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.376433 5601 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 17:33:55.376550 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.376541 5601 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 17:33:55.376578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.376555 5601 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 17:33:55.376578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.376562 5601 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 17:33:55.376578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.376568 5601 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 17:33:55.376578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.376573 5601 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 17:33:55.376578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.376578 5601 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 17:33:55.376792 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.376585 5601 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 17:33:55.376792 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.376590 5601 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 17:33:55.376792 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.376597 5601 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 17:33:55.376792 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.376609 5601 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 17:33:55.376792 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.376639 5601 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 17:33:55.376792 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.376649 5601 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 17:33:55.376792 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.376669 5601 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 17:33:55.376792 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.376674 5601 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 17:33:55.380157 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.380140 5601 server.go:1263] "Started kubelet" Nov 02 17:33:55.380259 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.380218 5601 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 17:33:55.380796 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.380373 5601 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 17:33:55.380796 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.380470 5601 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 17:33:55.380692 ip-10-0-141-169 systemd[1]: Started Kubernetes Kubelet. Nov 02 17:33:55.381524 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.381418 5601 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 17:33:55.381524 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.381518 5601 server.go:317] "Adding debug handlers to kubelet server" Nov 02 17:33:55.386894 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.386871 5601 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:33:55.387748 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.387070 5601 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 17:33:55.387748 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.387085 5601 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 17:33:55.387748 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.387148 5601 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:27:47 +0000 UTC" deadline="2028-04-21 16:17:55.763528105 +0000 UTC" Nov 02 17:33:55.387748 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.387165 5601 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12862h44m0.376365411s" Nov 02 17:33:55.387748 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.387174 5601 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 17:33:55.387748 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.387185 5601 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 17:33:55.387748 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.387253 5601 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 17:33:55.388758 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.388632 5601 factory.go:55] Registering systemd factory Nov 02 17:33:55.388758 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.388653 5601 factory.go:223] Registration of the systemd container factory successfully Nov 02 17:33:55.389745 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.389213 5601 factory.go:153] Registering CRI-O factory Nov 02 17:33:55.389745 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.389232 5601 factory.go:223] Registration of the crio container factory successfully Nov 02 17:33:55.389745 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.389307 5601 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 17:33:55.389745 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.389329 5601 factory.go:103] Registering Raw factory Nov 02 17:33:55.389745 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.389344 5601 manager.go:1199] Started watching for new ooms in manager Nov 02 17:33:55.390088 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.389800 5601 manager.go:319] Starting recovery of all containers Nov 02 17:33:55.390328 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.390305 5601 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:33:55.391225 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391192 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-kube-api-access-dfvxh" seLinuxMountContext="" Nov 02 17:33:55.391278 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391231 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ecf8c2f-987e-4441-a30d-2e75f3307aae" volumeName="kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 17:33:55.391278 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391245 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86286bdb-d69d-481a-b874-4ff3df42906f" volumeName="kubernetes.io/projected/86286bdb-d69d-481a-b874-4ff3df42906f-kube-api-access-lqxvs" seLinuxMountContext="" Nov 02 17:33:55.391278 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391259 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7" volumeName="kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret" seLinuxMountContext="" Nov 02 17:33:55.391278 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391271 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9300fb6-6926-4429-8797-603dfb361818" volumeName="kubernetes.io/projected/d9300fb6-6926-4429-8797-603dfb361818-kube-api-access-m4lln" seLinuxMountContext="" Nov 02 17:33:55.391391 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391284 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44174ea2-6704-423f-a176-c3abe64be85e" volumeName="kubernetes.io/projected/44174ea2-6704-423f-a176-c3abe64be85e-kube-api-access-95kp2" seLinuxMountContext="" Nov 02 17:33:55.391391 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391298 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a0753ab-8636-4ce9-9b3e-d89516b73c1c" volumeName="kubernetes.io/empty-dir/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-textfile" seLinuxMountContext="" Nov 02 17:33:55.391391 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391309 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a0753ab-8636-4ce9-9b3e-d89516b73c1c" volumeName="kubernetes.io/projected/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-kube-api-access-49hkf" seLinuxMountContext="" Nov 02 17:33:55.391391 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391322 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a0753ab-8636-4ce9-9b3e-d89516b73c1c" volumeName="kubernetes.io/secret/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 17:33:55.391391 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391334 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ecf8c2f-987e-4441-a30d-2e75f3307aae" volumeName="kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-grpc-tls" seLinuxMountContext="" Nov 02 17:33:55.391391 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391346 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d225fb4-f477-4971-bce1-8aaa24a79be8" volumeName="kubernetes.io/projected/8d225fb4-f477-4971-bce1-8aaa24a79be8-kube-api-access-kwnst" seLinuxMountContext="" Nov 02 17:33:55.391391 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391360 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9372be33-753d-4006-ba16-86fc44bd7c52" volumeName="kubernetes.io/secret/9372be33-753d-4006-ba16-86fc44bd7c52-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 17:33:55.391391 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391371 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a48715bf-e5aa-4b50-a9a5-4916e6fbefbc" volumeName="kubernetes.io/projected/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc-kube-api-access-4662z" seLinuxMountContext="" Nov 02 17:33:55.391391 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391382 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="065a4c3d-712b-4b65-b18f-0428e7a93672" volumeName="kubernetes.io/secret/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-state-metrics-tls" seLinuxMountContext="" Nov 02 17:33:55.391650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391396 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f379fbd-7dcf-4bff-a4c0-d946e711cf7e" volumeName="kubernetes.io/empty-dir/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-tmp" seLinuxMountContext="" Nov 02 17:33:55.391650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391406 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-config-out" seLinuxMountContext="" Nov 02 17:33:55.391650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391417 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ecf8c2f-987e-4441-a30d-2e75f3307aae" volumeName="kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-tls" seLinuxMountContext="" Nov 02 17:33:55.391650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391430 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8" volumeName="kubernetes.io/configmap/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-metrics-server-audit-profiles" seLinuxMountContext="" Nov 02 17:33:55.391650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391440 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8" volumeName="kubernetes.io/configmap/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 17:33:55.391650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391451 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 02 17:33:55.391650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391461 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="065a4c3d-712b-4b65-b18f-0428e7a93672" volumeName="kubernetes.io/secret/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 17:33:55.391650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391490 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e35ffc8-ef10-4532-af91-38841c3f4581" volumeName="kubernetes.io/configmap/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-daemon-config" seLinuxMountContext="" Nov 02 17:33:55.391650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391502 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 02 17:33:55.391650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391515 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8" volumeName="kubernetes.io/secret/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-secret-metrics-server-tls" seLinuxMountContext="" Nov 02 17:33:55.391650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391526 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ecf8c2f-987e-4441-a30d-2e75f3307aae" volumeName="kubernetes.io/projected/7ecf8c2f-987e-4441-a30d-2e75f3307aae-kube-api-access-tnz65" seLinuxMountContext="" Nov 02 17:33:55.391650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391539 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43f970d6-97ec-4e75-820b-aceefb7c37f5" volumeName="kubernetes.io/secret/43f970d6-97ec-4e75-820b-aceefb7c37f5-monitoring-plugin-cert" seLinuxMountContext="" Nov 02 17:33:55.391650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391552 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86286bdb-d69d-481a-b874-4ff3df42906f" volumeName="kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs" seLinuxMountContext="" Nov 02 17:33:55.391650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391564 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d225fb4-f477-4971-bce1-8aaa24a79be8" volumeName="kubernetes.io/configmap/8d225fb4-f477-4971-bce1-8aaa24a79be8-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 17:33:55.391650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391601 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9372be33-753d-4006-ba16-86fc44bd7c52" volumeName="kubernetes.io/empty-dir/9372be33-753d-4006-ba16-86fc44bd7c52-data-volume" seLinuxMountContext="" Nov 02 17:33:55.391650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391641 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ecf8c2f-987e-4441-a30d-2e75f3307aae" volumeName="kubernetes.io/empty-dir/7ecf8c2f-987e-4441-a30d-2e75f3307aae-tmp" seLinuxMountContext="" Nov 02 17:33:55.392060 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391656 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ecf8c2f-987e-4441-a30d-2e75f3307aae" volumeName="kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 02 17:33:55.392060 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391669 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a47b32de-3a53-4b34-8874-87049c4d747b" volumeName="kubernetes.io/secret/a47b32de-3a53-4b34-8874-87049c4d747b-cert" seLinuxMountContext="" Nov 02 17:33:55.392202 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.391681 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f379fbd-7dcf-4bff-a4c0-d946e711cf7e" volumeName="kubernetes.io/projected/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-kube-api-access-t97rb" seLinuxMountContext="" Nov 02 17:33:55.392234 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392211 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 02 17:33:55.392234 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392227 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-db" seLinuxMountContext="" Nov 02 17:33:55.392290 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392243 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-config" seLinuxMountContext="" Nov 02 17:33:55.392290 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392272 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ae7bace-4c4c-4f60-9863-5b2c29b54b76" volumeName="kubernetes.io/secret/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-openshift-state-metrics-tls" seLinuxMountContext="" Nov 02 17:33:55.392342 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392286 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ecf8c2f-987e-4441-a30d-2e75f3307aae" volumeName="kubernetes.io/configmap/7ecf8c2f-987e-4441-a30d-2e75f3307aae-metrics-client-ca" seLinuxMountContext="" Nov 02 17:33:55.392342 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392298 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d225fb4-f477-4971-bce1-8aaa24a79be8" volumeName="kubernetes.io/configmap/8d225fb4-f477-4971-bce1-8aaa24a79be8-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 17:33:55.392342 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392314 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c00d16e7-b92c-46c7-a68f-797274bbef93" volumeName="kubernetes.io/configmap/c00d16e7-b92c-46c7-a68f-797274bbef93-konnectivity-ca" seLinuxMountContext="" Nov 02 17:33:55.392342 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392328 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11bb6b24-b4e9-4764-a40a-e1daa2c613c2" volumeName="kubernetes.io/configmap/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-ovnkube-config" seLinuxMountContext="" Nov 02 17:33:55.392503 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392344 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11bb6b24-b4e9-4764-a40a-e1daa2c613c2" volumeName="kubernetes.io/configmap/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-env-overrides" seLinuxMountContext="" Nov 02 17:33:55.392503 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392358 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 02 17:33:55.392503 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392371 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ecf8c2f-987e-4441-a30d-2e75f3307aae" volumeName="kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 02 17:33:55.392503 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392384 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afbca029-7d59-4cfb-b08c-353e174b0355" volumeName="kubernetes.io/secret/afbca029-7d59-4cfb-b08c-353e174b0355-metrics-tls" seLinuxMountContext="" Nov 02 17:33:55.392503 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392400 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8" volumeName="kubernetes.io/secret/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 02 17:33:55.392503 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392416 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="065a4c3d-712b-4b65-b18f-0428e7a93672" volumeName="kubernetes.io/configmap/065a4c3d-712b-4b65-b18f-0428e7a93672-metrics-client-ca" seLinuxMountContext="" Nov 02 17:33:55.392503 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392431 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="065a4c3d-712b-4b65-b18f-0428e7a93672" volumeName="kubernetes.io/empty-dir/065a4c3d-712b-4b65-b18f-0428e7a93672-volume-directive-shadow" seLinuxMountContext="" Nov 02 17:33:55.392503 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392444 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a0753ab-8636-4ce9-9b3e-d89516b73c1c" volumeName="kubernetes.io/configmap/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-metrics-client-ca" seLinuxMountContext="" Nov 02 17:33:55.392503 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392457 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-tls-assets" seLinuxMountContext="" Nov 02 17:33:55.392503 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392470 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 02 17:33:55.392503 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392484 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ae7bace-4c4c-4f60-9863-5b2c29b54b76" volumeName="kubernetes.io/secret/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 17:33:55.392503 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392497 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8" volumeName="kubernetes.io/projected/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-kube-api-access-dpp8l" seLinuxMountContext="" Nov 02 17:33:55.393080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392510 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e35ffc8-ef10-4532-af91-38841c3f4581" volumeName="kubernetes.io/projected/3e35ffc8-ef10-4532-af91-38841c3f4581-kube-api-access-mlzk8" seLinuxMountContext="" Nov 02 17:33:55.393080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392522 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44174ea2-6704-423f-a176-c3abe64be85e" volumeName="kubernetes.io/configmap/44174ea2-6704-423f-a176-c3abe64be85e-iptables-alerter-script" seLinuxMountContext="" Nov 02 17:33:55.393080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392537 5601 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:33:55.393080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392540 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-metrics-client-ca" seLinuxMountContext="" Nov 02 17:33:55.393080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392886 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 02 17:33:55.393080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392911 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afbca029-7d59-4cfb-b08c-353e174b0355" volumeName="kubernetes.io/configmap/afbca029-7d59-4cfb-b08c-353e174b0355-config-volume" seLinuxMountContext="" Nov 02 17:33:55.393080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392926 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9300fb6-6926-4429-8797-603dfb361818" volumeName="kubernetes.io/configmap/d9300fb6-6926-4429-8797-603dfb361818-serviceca" seLinuxMountContext="" Nov 02 17:33:55.393080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392940 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="065a4c3d-712b-4b65-b18f-0428e7a93672" volumeName="kubernetes.io/configmap/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 02 17:33:55.393080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392954 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11bb6b24-b4e9-4764-a40a-e1daa2c613c2" volumeName="kubernetes.io/configmap/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-ovnkube-script-lib" seLinuxMountContext="" Nov 02 17:33:55.393080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392966 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-web-config" seLinuxMountContext="" Nov 02 17:33:55.393080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392979 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ecf8c2f-987e-4441-a30d-2e75f3307aae" volumeName="kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 02 17:33:55.393080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.392995 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ce34815-aef2-487a-afd1-fe81200a423d" volumeName="kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr" seLinuxMountContext="" Nov 02 17:33:55.393080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393008 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a48715bf-e5aa-4b50-a9a5-4916e6fbefbc" volumeName="kubernetes.io/empty-dir/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc-tmp-dir" seLinuxMountContext="" Nov 02 17:33:55.393080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393021 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afbca029-7d59-4cfb-b08c-353e174b0355" volumeName="kubernetes.io/projected/afbca029-7d59-4cfb-b08c-353e174b0355-kube-api-access-hdccp" seLinuxMountContext="" Nov 02 17:33:55.393080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393034 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="065a4c3d-712b-4b65-b18f-0428e7a93672" volumeName="kubernetes.io/projected/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-api-access-k8lqk" seLinuxMountContext="" Nov 02 17:33:55.393080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393046 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 02 17:33:55.393080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393062 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-grpc-tls" seLinuxMountContext="" Nov 02 17:33:55.393080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393078 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ae7bace-4c4c-4f60-9863-5b2c29b54b76" volumeName="kubernetes.io/configmap/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-metrics-client-ca" seLinuxMountContext="" Nov 02 17:33:55.393080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393091 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a47b32de-3a53-4b34-8874-87049c4d747b" volumeName="kubernetes.io/projected/a47b32de-3a53-4b34-8874-87049c4d747b-kube-api-access-7vjcv" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393104 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ae7bace-4c4c-4f60-9863-5b2c29b54b76" volumeName="kubernetes.io/projected/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-kube-api-access-pqn2p" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393117 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f379fbd-7dcf-4bff-a4c0-d946e711cf7e" volumeName="kubernetes.io/empty-dir/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-tuned" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393129 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a0753ab-8636-4ce9-9b3e-d89516b73c1c" volumeName="kubernetes.io/secret/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-tls" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393143 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11bb6b24-b4e9-4764-a40a-e1daa2c613c2" volumeName="kubernetes.io/secret/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393156 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393168 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393182 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-metrics-client-certs" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393199 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393214 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a0753ab-8636-4ce9-9b3e-d89516b73c1c" volumeName="kubernetes.io/configmap/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393232 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11bb6b24-b4e9-4764-a40a-e1daa2c613c2" volumeName="kubernetes.io/projected/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-kube-api-access-cnt46" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393245 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ae9769-93f1-4962-b68c-22f5b4658484" volumeName="kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393260 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d225fb4-f477-4971-bce1-8aaa24a79be8" volumeName="kubernetes.io/configmap/8d225fb4-f477-4971-bce1-8aaa24a79be8-cni-binary-copy" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393272 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9372be33-753d-4006-ba16-86fc44bd7c52" volumeName="kubernetes.io/projected/9372be33-753d-4006-ba16-86fc44bd7c52-kube-api-access-rpg54" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393285 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afbca029-7d59-4cfb-b08c-353e174b0355" volumeName="kubernetes.io/empty-dir/afbca029-7d59-4cfb-b08c-353e174b0355-tmp-dir" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393299 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8" volumeName="kubernetes.io/secret/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-client-ca-bundle" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393315 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40e6a790-de24-49f1-b507-ed74d1446868" volumeName="kubernetes.io/projected/40e6a790-de24-49f1-b507-ed74d1446868-kube-api-access-ptckf" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393331 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9372be33-753d-4006-ba16-86fc44bd7c52" volumeName="kubernetes.io/configmap/9372be33-753d-4006-ba16-86fc44bd7c52-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393345 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c00d16e7-b92c-46c7-a68f-797274bbef93" volumeName="kubernetes.io/secret/c00d16e7-b92c-46c7-a68f-797274bbef93-agent-certs" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393359 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8" volumeName="kubernetes.io/empty-dir/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-audit-log" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393371 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e35ffc8-ef10-4532-af91-38841c3f4581" volumeName="kubernetes.io/configmap/3e35ffc8-ef10-4532-af91-38841c3f4581-cni-binary-copy" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393386 5601 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43f970d6-97ec-4e75-820b-aceefb7c37f5" volumeName="kubernetes.io/projected/43f970d6-97ec-4e75-820b-aceefb7c37f5-kube-api-access-wwl2c" seLinuxMountContext="" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393402 5601 reconstruct.go:97] "Volume reconstruction finished" Nov 02 17:33:55.393820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.393411 5601 reconciler.go:29] "Reconciler: start to sync state" Nov 02 17:33:55.397950 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.397928 5601 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 17:33:55.416723 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.416641 5601 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 17:33:55.418044 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.418021 5601 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 17:33:55.418044 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.418042 5601 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 17:33:55.418212 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.418079 5601 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 17:33:55.418212 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:33:55.418129 5601 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 17:33:55.422129 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.422108 5601 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:33:55.434220 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.434190 5601 generic.go:356] "Generic (PLEG): container finished" podID="ca96163b2bc1e65e3d055163a6272914" containerID="3d5c1447bb3db591bd57f2cb5192f39f60f495a403bbfa9aa442646e744abaf3" exitCode=0 Nov 02 17:33:55.452188 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.452162 5601 generic.go:356] "Generic (PLEG): container finished" podID="8d225fb4-f477-4971-bce1-8aaa24a79be8" containerID="3c5d77d76c80b259b3a53fb8e0951965ee0149607f86de4d2b355a2aa64caf41" exitCode=0 Nov 02 17:33:55.452188 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.452184 5601 generic.go:356] "Generic (PLEG): container finished" podID="8d225fb4-f477-4971-bce1-8aaa24a79be8" containerID="77819e1f88605c92d8fda1bf242fba510f367e34da769b2d6b921d7b94f30270" exitCode=0 Nov 02 17:33:55.452188 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.452191 5601 generic.go:356] "Generic (PLEG): container finished" podID="8d225fb4-f477-4971-bce1-8aaa24a79be8" containerID="0ddbc79f333f6590139d6461f85e401600eb42b4c7a2f9a7b15e57fcab327e5a" exitCode=0 Nov 02 17:33:55.452380 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.452197 5601 generic.go:356] "Generic (PLEG): container finished" podID="8d225fb4-f477-4971-bce1-8aaa24a79be8" containerID="14846034d093b9c9eaa4f86453671cb7514e2858b0960aa6afb3f48a5e2c211e" exitCode=0 Nov 02 17:33:55.452380 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.452204 5601 generic.go:356] "Generic (PLEG): container finished" podID="8d225fb4-f477-4971-bce1-8aaa24a79be8" containerID="e82ced2864ae60849035add8cd5469380c1f855ab50b79038a475e5fd992c184" exitCode=0 Nov 02 17:33:55.452380 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.452209 5601 generic.go:356] "Generic (PLEG): container finished" podID="8d225fb4-f477-4971-bce1-8aaa24a79be8" containerID="8f1b6613840c3b99479194e69f1c0929c4f066d5d03d12f80f650d44b1cabebd" exitCode=0 Nov 02 17:33:55.469877 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.469853 5601 generic.go:356] "Generic (PLEG): container finished" podID="54ae9769-93f1-4962-b68c-22f5b4658484" containerID="7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87" exitCode=0 Nov 02 17:33:55.472067 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.472046 5601 generic.go:356] "Generic (PLEG): container finished" podID="0a0753ab-8636-4ce9-9b3e-d89516b73c1c" containerID="fb2560e050014c6afba8c9df47b9b554fbc9bce0e32d564539912e18e5d2a04e" exitCode=0 Nov 02 17:33:55.504673 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.504645 5601 manager.go:324] Recovery completed Nov 02 17:33:55.519186 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:33:55.519125 5601 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 17:33:55.526044 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.526028 5601 cpu_manager.go:226] "Starting" policy="none" Nov 02 17:33:55.526044 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.526041 5601 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 17:33:55.526166 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.526058 5601 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 17:33:55.526196 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.526168 5601 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 17:33:55.526196 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.526179 5601 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 17:33:55.526255 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.526196 5601 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 17:33:55.526255 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.526202 5601 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 17:33:55.526255 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.526207 5601 policy_none.go:50] "Start" Nov 02 17:33:55.526255 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.526216 5601 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 17:33:55.526255 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.526225 5601 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 17:33:55.526382 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.526339 5601 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 17:33:55.526382 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.526346 5601 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 17:33:55.526382 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.526352 5601 policy_none.go:44] "Start" Nov 02 17:33:55.531007 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.530982 5601 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 17:33:55.534080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.534063 5601 manager.go:341] "Starting Device Plugin manager" Nov 02 17:33:55.534169 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:33:55.534088 5601 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 17:33:55.534169 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.534097 5601 server.go:87] "Starting device plugin registration server" Nov 02 17:33:55.534788 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.534771 5601 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 17:33:55.534876 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.534785 5601 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 17:33:55.534931 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.534889 5601 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 17:33:55.535071 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.535033 5601 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 17:33:55.535406 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.535383 5601 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 17:33:55.535406 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.535405 5601 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 17:33:55.535593 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.535482 5601 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:33:55.535067125Z","UUID":"8d30ef18-1822-4968-8f5d-d2f9cc85ab00","Handler":null,"Name":"","Endpoint":""} Nov 02 17:33:55.537337 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.537256 5601 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 17:33:55.537435 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.537357 5601 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 17:33:55.537435 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:33:55.537390 5601 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 17:33:55.635830 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.635793 5601 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:33:55.637546 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.637531 5601 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-169.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:33:55.637607 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.637559 5601 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-169.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:33:55.637607 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.637569 5601 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-169.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:33:55.637755 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.637609 5601 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:55.653717 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.653697 5601 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:55.653768 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.653749 5601 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:55.719506 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.719464 5601 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal"] Nov 02 17:33:55.720151 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.720103 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" event={"ID":"ca96163b2bc1e65e3d055163a6272914","Type":"ContainerStarted","Data":"e98b36ae1bfaf2e9af9278394c6aae99ae76ed9d74424268d5a56c599de5ddd2"} Nov 02 17:33:55.720200 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.720163 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" event={"ID":"ca96163b2bc1e65e3d055163a6272914","Type":"ContainerDied","Data":"3d5c1447bb3db591bd57f2cb5192f39f60f495a403bbfa9aa442646e744abaf3"} Nov 02 17:33:55.720200 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.720179 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" event={"ID":"ca96163b2bc1e65e3d055163a6272914","Type":"ContainerStarted","Data":"7c97a3b904cb02796a1e696cc3a9a28c0baa6b091b3366784a6b7c8c039a7488"} Nov 02 17:33:55.720274 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.720142 5601 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:55.720274 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.720244 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal" event={"ID":"f82e5ebcf2f8d3b5e95e3af83d2365db","Type":"ContainerStarted","Data":"2f70519c3749ca0c14bcc25d82701ef9b60acbaeb2b43bb7ff828d99fd572d32"} Nov 02 17:33:55.720274 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.720264 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal" event={"ID":"f82e5ebcf2f8d3b5e95e3af83d2365db","Type":"ContainerStarted","Data":"c1cacef195d5be9ba06c4120bc166fa099b012a5b5e9eadce24a287821c1064f"} Nov 02 17:33:55.720374 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.720308 5601 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:55.731551 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.731532 5601 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:33:55.731604 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:33:55.731567 5601 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:55.732311 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.732299 5601 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:33:55.732352 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:33:55.732322 5601 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:55.799699 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.799638 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca96163b2bc1e65e3d055163a6272914-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal\" (UID: \"ca96163b2bc1e65e3d055163a6272914\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:55.799699 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.799670 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca96163b2bc1e65e3d055163a6272914-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal\" (UID: \"ca96163b2bc1e65e3d055163a6272914\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:55.799699 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.799689 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f82e5ebcf2f8d3b5e95e3af83d2365db-config\") pod \"kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal\" (UID: \"f82e5ebcf2f8d3b5e95e3af83d2365db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:55.900230 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.900199 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca96163b2bc1e65e3d055163a6272914-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal\" (UID: \"ca96163b2bc1e65e3d055163a6272914\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:55.900337 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.900238 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca96163b2bc1e65e3d055163a6272914-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal\" (UID: \"ca96163b2bc1e65e3d055163a6272914\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:55.900337 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.900257 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f82e5ebcf2f8d3b5e95e3af83d2365db-config\") pod \"kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal\" (UID: \"f82e5ebcf2f8d3b5e95e3af83d2365db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:55.900337 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.900298 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f82e5ebcf2f8d3b5e95e3af83d2365db-config\") pod \"kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal\" (UID: \"f82e5ebcf2f8d3b5e95e3af83d2365db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:55.900337 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.900314 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca96163b2bc1e65e3d055163a6272914-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal\" (UID: \"ca96163b2bc1e65e3d055163a6272914\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:55.900454 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:55.900314 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca96163b2bc1e65e3d055163a6272914-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal\" (UID: \"ca96163b2bc1e65e3d055163a6272914\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:56.373143 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.373111 5601 apiserver.go:52] "Watching apiserver" Nov 02 17:33:56.381975 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.381945 5601 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 17:33:56.382518 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.382493 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0","openshift-monitoring/monitoring-plugin-77f4f76594-4c892","kube-system/global-pull-secret-syncer-f4xwb","openshift-ingress-canary/ingress-canary-nwbhl","openshift-cluster-node-tuning-operator/tuned-s9kl9","openshift-network-diagnostics/network-check-target-424d5","kube-system/konnectivity-agent-cjnbh","openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477","openshift-monitoring/thanos-querier-6f98f548cc-srzfq","openshift-multus/multus-gbc4r","openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph","openshift-monitoring/metrics-server-dcf4d45cc-49kqs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2","openshift-multus/multus-additional-cni-plugins-vdll2","openshift-network-operator/iptables-alerter-5l7cq","openshift-ovn-kubernetes/ovnkube-node-5wh7f","openshift-image-registry/node-ca-q28kf","openshift-multus/network-metrics-daemon-dfthz","kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal","openshift-dns/dns-default-rtxf5","openshift-dns/node-resolver-5tkc8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal","openshift-monitoring/node-exporter-8dtcc","openshift-insights/insights-runtime-extractor-vrm5q"] Nov 02 17:33:56.386689 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.386667 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jl4vs\"" Nov 02 17:33:56.387006 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.386985 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 17:33:56.387065 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.387027 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dcw65\"" Nov 02 17:33:56.387065 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.387037 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lkfrw\"" Nov 02 17:33:56.387223 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.387206 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 17:33:56.387657 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.387636 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ttfxd\"" Nov 02 17:33:56.387962 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.387943 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-52q82\"" Nov 02 17:33:56.388049 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.388028 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:33:56.388103 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.388076 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 17:33:56.388177 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.388104 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 17:33:56.388177 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.388154 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 17:33:56.388177 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.388157 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 17:33:56.388454 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.388251 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 17:33:56.388454 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.388252 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 17:33:56.388454 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.388264 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 17:33:56.388454 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.388289 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xjkc9\"" Nov 02 17:33:56.388454 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.388325 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 17:33:56.388454 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.388326 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 17:33:56.388454 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.388361 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 17:33:56.388741 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.388462 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 17:33:56.388885 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.388862 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 17:33:56.389103 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.389086 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 17:33:56.389372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.389355 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 17:33:56.389372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.389368 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sbqnj\"" Nov 02 17:33:56.389372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.389380 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 17:33:56.389564 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.389437 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 17:33:56.389704 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.389687 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 17:33:56.389775 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.389764 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 17:33:56.389841 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.389776 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:33:56.389841 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.389779 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jmxfr\"" Nov 02 17:33:56.389841 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.389802 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 17:33:56.389841 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.389824 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k78hz\"" Nov 02 17:33:56.389975 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.389892 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 17:33:56.390004 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.389991 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 17:33:56.390050 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.390039 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zg2g5\"" Nov 02 17:33:56.390333 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.390321 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 17:33:56.390681 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.390661 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 17:33:56.390681 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.390671 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 17:33:56.390812 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.390784 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dtld9\"" Nov 02 17:33:56.390852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.390818 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 17:33:56.391710 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.391692 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wjf7q\"" Nov 02 17:33:56.391855 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.391802 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 17:33:56.391954 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.391893 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 17:33:56.392469 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.392450 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 17:33:56.392529 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.392456 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 17:33:56.392845 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.392829 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 17:33:56.392951 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.392933 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 17:33:56.393174 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.393158 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 17:33:56.393452 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.393434 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dp4wq\"" Nov 02 17:33:56.393586 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.393440 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vxnnv\"" Nov 02 17:33:56.394000 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.393980 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 17:33:56.394109 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.394059 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 17:33:56.394291 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.394272 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 17:33:56.394347 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.394289 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9wn4g\"" Nov 02 17:33:56.394393 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.394354 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 17:33:56.394601 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.394586 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 17:33:56.395015 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.394992 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 17:33:56.395212 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.395193 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 17:33:56.395396 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.395376 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jdltr\"" Nov 02 17:33:56.395485 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.395469 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 17:33:56.395548 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.395493 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 17:33:56.396868 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.396846 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 17:33:56.397074 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.397060 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6p4gz\"" Nov 02 17:33:56.397141 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.397130 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 17:33:56.397193 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.397183 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 17:33:56.402672 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.402651 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4lln\" (UniqueName: \"kubernetes.io/projected/d9300fb6-6926-4429-8797-603dfb361818-kube-api-access-m4lln\") pod \"node-ca-q28kf\" (UID: \"d9300fb6-6926-4429-8797-603dfb361818\") " pod="openshift-image-registry/node-ca-q28kf" Nov 02 17:33:56.402771 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.402679 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc-tmp-dir\") pod \"node-resolver-5tkc8\" (UID: \"a48715bf-e5aa-4b50-a9a5-4916e6fbefbc\") " pod="openshift-dns/node-resolver-5tkc8" Nov 02 17:33:56.402771 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.402698 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-socket-dir-parent\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.402771 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.402729 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-run-netns\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.402917 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.402799 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc-tmp-dir\") pod \"node-resolver-5tkc8\" (UID: \"a48715bf-e5aa-4b50-a9a5-4916e6fbefbc\") " pod="openshift-dns/node-resolver-5tkc8" Nov 02 17:33:56.402917 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.402823 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-conf-dir\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.402917 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.402898 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d9300fb6-6926-4429-8797-603dfb361818-serviceca\") pod \"node-ca-q28kf\" (UID: \"d9300fb6-6926-4429-8797-603dfb361818\") " pod="openshift-image-registry/node-ca-q28kf" Nov 02 17:33:56.403099 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.402938 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tmr\" (UniqueName: \"kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr\") pod \"network-check-target-424d5\" (UID: \"9ce34815-aef2-487a-afd1-fe81200a423d\") " pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:56.403099 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.402995 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-ovn-node-metrics-cert\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.403099 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403031 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-run-multus-certs\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.403099 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403057 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-sysctl-conf\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.403236 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403100 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-sys\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.403236 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403163 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d225fb4-f477-4971-bce1-8aaa24a79be8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:33:56.403236 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403199 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d225fb4-f477-4971-bce1-8aaa24a79be8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:33:56.403378 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403234 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4662z\" (UniqueName: \"kubernetes.io/projected/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc-kube-api-access-4662z\") pod \"node-resolver-5tkc8\" (UID: \"a48715bf-e5aa-4b50-a9a5-4916e6fbefbc\") " pod="openshift-dns/node-resolver-5tkc8" Nov 02 17:33:56.403378 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403239 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 17:33:56.403378 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403303 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-systemd-units\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.403378 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403335 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-run-k8s-cni-cncf-io\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.403539 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403389 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-hostroot\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.403539 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403400 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 17:33:56.403539 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403410 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-ovn-node-metrics-cert\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.403539 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403444 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-kubernetes\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.403539 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403455 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d225fb4-f477-4971-bce1-8aaa24a79be8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:33:56.403539 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403476 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-etc-openvswitch\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.403539 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403525 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnt46\" (UniqueName: \"kubernetes.io/projected/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-kube-api-access-cnt46\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.403890 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403550 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d9300fb6-6926-4429-8797-603dfb361818-serviceca\") pod \"node-ca-q28kf\" (UID: \"d9300fb6-6926-4429-8797-603dfb361818\") " pod="openshift-image-registry/node-ca-q28kf" Nov 02 17:33:56.403890 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403561 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-tuned\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.403890 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403673 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-tuned\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.403890 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403673 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c00d16e7-b92c-46c7-a68f-797274bbef93-agent-certs\") pod \"konnectivity-agent-cjnbh\" (UID: \"c00d16e7-b92c-46c7-a68f-797274bbef93\") " pod="kube-system/konnectivity-agent-cjnbh" Nov 02 17:33:56.403890 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403751 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-slash\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.403890 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403768 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-run\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.403890 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403785 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d225fb4-f477-4971-bce1-8aaa24a79be8-system-cni-dir\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:33:56.403890 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403850 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d225fb4-f477-4971-bce1-8aaa24a79be8-cnibin\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:33:56.403890 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403884 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.403890 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403889 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c00d16e7-b92c-46c7-a68f-797274bbef93-agent-certs\") pod \"konnectivity-agent-cjnbh\" (UID: \"c00d16e7-b92c-46c7-a68f-797274bbef93\") " pod="kube-system/konnectivity-agent-cjnbh" Nov 02 17:33:56.404372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403936 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-dbus\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:56.404372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.403983 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-run-netns\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.404372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404021 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-run-systemd\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.404372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404054 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-run-ovn\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.404372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404067 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 17:33:56.404372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404080 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-var-lib-cni-bin\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.404372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404109 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptckf\" (UniqueName: \"kubernetes.io/projected/40e6a790-de24-49f1-b507-ed74d1446868-kube-api-access-ptckf\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.404372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404159 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-var-lib-openvswitch\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.404372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404181 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 17:33:56.404372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404183 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-node-log\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.404372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404230 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.404372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404260 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-var-lib-kubelet\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.404372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404262 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 17:33:56.404372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404298 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlzk8\" (UniqueName: \"kubernetes.io/projected/3e35ffc8-ef10-4532-af91-38841c3f4581-kube-api-access-mlzk8\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.404372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404326 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-socket-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.404372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404352 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-kubelet-config\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:56.404372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404377 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404366 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404442 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44174ea2-6704-423f-a176-c3abe64be85e-host-slash\") pod \"iptables-alerter-5l7cq\" (UID: \"44174ea2-6704-423f-a176-c3abe64be85e\") " pod="openshift-network-operator/iptables-alerter-5l7cq" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404484 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-daemon-config\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404515 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-sysctl-d\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404525 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zrqks\"" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404571 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d225fb4-f477-4971-bce1-8aaa24a79be8-cni-binary-copy\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404591 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc-hosts-file\") pod \"node-resolver-5tkc8\" (UID: \"a48715bf-e5aa-4b50-a9a5-4916e6fbefbc\") " pod="openshift-dns/node-resolver-5tkc8" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404653 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-ovnkube-script-lib\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404677 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs\") pod \"network-metrics-daemon-dfthz\" (UID: \"86286bdb-d69d-481a-b874-4ff3df42906f\") " pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404677 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404698 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95kp2\" (UniqueName: \"kubernetes.io/projected/44174ea2-6704-423f-a176-c3abe64be85e-kube-api-access-95kp2\") pod \"iptables-alerter-5l7cq\" (UID: \"44174ea2-6704-423f-a176-c3abe64be85e\") " pod="openshift-network-operator/iptables-alerter-5l7cq" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404715 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-daemon-config\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404774 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d225fb4-f477-4971-bce1-8aaa24a79be8-cni-binary-copy\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404789 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404820 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404721 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-etc-kubernetes\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404896 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-sysconfig\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.404932 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-systemd\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405011 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-kubelet\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405024 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86286bdb-d69d-481a-b874-4ff3df42906f-metrics-certs\") pod \"network-metrics-daemon-dfthz\" (UID: \"86286bdb-d69d-481a-b874-4ff3df42906f\") " pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:56.405058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405013 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-ovnkube-script-lib\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405123 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-env-overrides\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405194 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-var-lib-kubelet\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405248 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-tmp\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405269 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c00d16e7-b92c-46c7-a68f-797274bbef93-konnectivity-ca\") pod \"konnectivity-agent-cjnbh\" (UID: \"c00d16e7-b92c-46c7-a68f-797274bbef93\") " pod="kube-system/konnectivity-agent-cjnbh" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405272 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405301 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-device-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405308 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405324 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-etc-selinux\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405351 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405388 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-ovnkube-config\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405418 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-log-socket\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405413 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-tmp\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405438 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-env-overrides\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405467 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqxvs\" (UniqueName: \"kubernetes.io/projected/86286bdb-d69d-481a-b874-4ff3df42906f-kube-api-access-lqxvs\") pod \"network-metrics-daemon-dfthz\" (UID: \"86286bdb-d69d-481a-b874-4ff3df42906f\") " pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405493 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-var-lib-cni-multus\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405520 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-lib-modules\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405565 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c00d16e7-b92c-46c7-a68f-797274bbef93-konnectivity-ca\") pod \"konnectivity-agent-cjnbh\" (UID: \"c00d16e7-b92c-46c7-a68f-797274bbef93\") " pod="kube-system/konnectivity-agent-cjnbh" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405581 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-ovnkube-config\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405601 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t97rb\" (UniqueName: \"kubernetes.io/projected/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-kube-api-access-t97rb\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405598 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405670 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405676 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405683 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405684 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405698 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405713 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405713 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9300fb6-6926-4429-8797-603dfb361818-host\") pod \"node-ca-q28kf\" (UID: \"d9300fb6-6926-4429-8797-603dfb361818\") " pod="openshift-image-registry/node-ca-q28kf" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405816 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwnst\" (UniqueName: \"kubernetes.io/projected/8d225fb4-f477-4971-bce1-8aaa24a79be8-kube-api-access-kwnst\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405822 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-kz9hl\"" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405866 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405871 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405876 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wrf7q\"" Nov 02 17:33:56.405889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405892 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-sys-fs\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405934 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405940 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wsqql\"" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405996 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/44174ea2-6704-423f-a176-c3abe64be85e-iptables-alerter-script\") pod \"iptables-alerter-5l7cq\" (UID: \"44174ea2-6704-423f-a176-c3abe64be85e\") " pod="openshift-network-operator/iptables-alerter-5l7cq" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.405997 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406017 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-cni-dir\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406043 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-cnibin\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406022 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bjc4qab0pol8j\"" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406069 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e35ffc8-ef10-4532-af91-38841c3f4581-cni-binary-copy\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406078 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406091 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-host\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406118 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-cni-bin\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406134 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cs83l4avv8s1v\"" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406136 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fr655\"" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406143 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406172 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406175 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/44174ea2-6704-423f-a176-c3abe64be85e-iptables-alerter-script\") pod \"iptables-alerter-5l7cq\" (UID: \"44174ea2-6704-423f-a176-c3abe64be85e\") " pod="openshift-network-operator/iptables-alerter-5l7cq" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406188 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-modprobe-d\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406260 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d225fb4-f477-4971-bce1-8aaa24a79be8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406297 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-registration-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406338 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-run-openvswitch\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406358 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e35ffc8-ef10-4532-af91-38841c3f4581-cni-binary-copy\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406391 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-cni-netd\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406464 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-original-pull-secret\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406509 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f7cgovttces7q\"" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406578 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d225fb4-f477-4971-bce1-8aaa24a79be8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:33:56.407505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.406717 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-system-cni-dir\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.408938 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.408470 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 17:33:56.408938 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.408780 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 17:33:56.410378 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.410361 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4662z\" (UniqueName: \"kubernetes.io/projected/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc-kube-api-access-4662z\") pod \"node-resolver-5tkc8\" (UID: \"a48715bf-e5aa-4b50-a9a5-4916e6fbefbc\") " pod="openshift-dns/node-resolver-5tkc8" Nov 02 17:33:56.410900 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.410881 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlzk8\" (UniqueName: \"kubernetes.io/projected/3e35ffc8-ef10-4532-af91-38841c3f4581-kube-api-access-mlzk8\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.411368 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.411346 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4lln\" (UniqueName: \"kubernetes.io/projected/d9300fb6-6926-4429-8797-603dfb361818-kube-api-access-m4lln\") pod \"node-ca-q28kf\" (UID: \"d9300fb6-6926-4429-8797-603dfb361818\") " pod="openshift-image-registry/node-ca-q28kf" Nov 02 17:33:56.411428 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.411369 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9tmr\" (UniqueName: \"kubernetes.io/projected/9ce34815-aef2-487a-afd1-fe81200a423d-kube-api-access-m9tmr\") pod \"network-check-target-424d5\" (UID: \"9ce34815-aef2-487a-afd1-fe81200a423d\") " pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:33:56.411670 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.411650 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnt46\" (UniqueName: \"kubernetes.io/projected/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-kube-api-access-cnt46\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.411874 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.411856 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqxvs\" (UniqueName: \"kubernetes.io/projected/86286bdb-d69d-481a-b874-4ff3df42906f-kube-api-access-lqxvs\") pod \"network-metrics-daemon-dfthz\" (UID: \"86286bdb-d69d-481a-b874-4ff3df42906f\") " pod="openshift-multus/network-metrics-daemon-dfthz" Nov 02 17:33:56.411928 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.411873 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95kp2\" (UniqueName: \"kubernetes.io/projected/44174ea2-6704-423f-a176-c3abe64be85e-kube-api-access-95kp2\") pod \"iptables-alerter-5l7cq\" (UID: \"44174ea2-6704-423f-a176-c3abe64be85e\") " pod="openshift-network-operator/iptables-alerter-5l7cq" Nov 02 17:33:56.412001 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.411987 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptckf\" (UniqueName: \"kubernetes.io/projected/40e6a790-de24-49f1-b507-ed74d1446868-kube-api-access-ptckf\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.412481 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.412465 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t97rb\" (UniqueName: \"kubernetes.io/projected/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-kube-api-access-t97rb\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.412735 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.412717 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwnst\" (UniqueName: \"kubernetes.io/projected/8d225fb4-f477-4971-bce1-8aaa24a79be8-kube-api-access-kwnst\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:33:56.485108 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.485082 5601 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:56.485554 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.485537 5601 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:56.488663 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.488641 5601 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 17:33:56.497193 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.497154 5601 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:33:56.497330 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:33:56.497268 5601 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:56.497823 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.497807 5601 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:33:56.497913 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:33:56.497840 5601 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.us-east-2.compute.internal" Nov 02 17:33:56.506934 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.506915 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-slash\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.507033 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.506942 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwl2c\" (UniqueName: \"kubernetes.io/projected/43f970d6-97ec-4e75-820b-aceefb7c37f5-kube-api-access-wwl2c\") pod \"monitoring-plugin-77f4f76594-4c892\" (UID: \"43f970d6-97ec-4e75-820b-aceefb7c37f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4c892" Nov 02 17:33:56.507033 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.506958 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-var-lib-kubelet\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.507033 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.506977 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-secret-metrics-server-client-certs\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:56.507033 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507000 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.507232 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507045 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-dbus\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:56.507232 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507068 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-slash\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.507232 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507086 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-run-netns\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.507232 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507112 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-run-systemd\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.507232 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507117 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-var-lib-kubelet\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.507232 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507133 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-dbus\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:56.507232 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507137 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-node-log\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.507232 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507163 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-run-netns\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.507232 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507163 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.507232 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507173 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.507232 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507190 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/065a4c3d-712b-4b65-b18f-0428e7a93672-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:56.507232 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507200 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-node-log\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.507232 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507217 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.507232 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507217 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9372be33-753d-4006-ba16-86fc44bd7c52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507233 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-run-systemd\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507251 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpg54\" (UniqueName: \"kubernetes.io/projected/9372be33-753d-4006-ba16-86fc44bd7c52-kube-api-access-rpg54\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507256 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-secret-metrics-server-client-certs\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507288 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-socket-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507310 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-kubelet-config\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507326 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44174ea2-6704-423f-a176-c3abe64be85e-host-slash\") pod \"iptables-alerter-5l7cq\" (UID: \"44174ea2-6704-423f-a176-c3abe64be85e\") " pod="openshift-network-operator/iptables-alerter-5l7cq" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507344 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-sysctl-d\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507361 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-socket-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507369 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/065a4c3d-712b-4b65-b18f-0428e7a93672-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507383 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7-kubelet-config\") pod \"global-pull-secret-syncer-f4xwb\" (UID: \"d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7\") " pod="kube-system/global-pull-secret-syncer-f4xwb" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507395 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44174ea2-6704-423f-a176-c3abe64be85e-host-slash\") pod \"iptables-alerter-5l7cq\" (UID: \"44174ea2-6704-423f-a176-c3abe64be85e\") " pod="openshift-network-operator/iptables-alerter-5l7cq" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507441 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/065a4c3d-712b-4b65-b18f-0428e7a93672-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507443 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9372be33-753d-4006-ba16-86fc44bd7c52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507470 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-config\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507481 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-sysctl-d\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507497 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507545 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507572 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507546 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/065a4c3d-712b-4b65-b18f-0428e7a93672-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507601 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507671 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-kubelet\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507694 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-config\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507706 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-tls\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507725 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507732 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-kubelet\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507734 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a47b32de-3a53-4b34-8874-87049c4d747b-cert\") pod \"ingress-canary-nwbhl\" (UID: \"a47b32de-3a53-4b34-8874-87049c4d747b\") " pod="openshift-ingress-canary/ingress-canary-nwbhl" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507769 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-client-ca-bundle\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507796 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507810 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507821 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-grpc-tls\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507793 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507846 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507847 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-var-lib-cni-multus\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.507862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507878 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpp8l\" (UniqueName: \"kubernetes.io/projected/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-kube-api-access-dpp8l\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507896 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507895 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-var-lib-cni-multus\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507925 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-tls\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507926 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a47b32de-3a53-4b34-8874-87049c4d747b-cert\") pod \"ingress-canary-nwbhl\" (UID: \"a47b32de-3a53-4b34-8874-87049c4d747b\") " pod="openshift-ingress-canary/ingress-canary-nwbhl" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507953 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-cni-bin\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507974 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-secret-metrics-server-tls\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507990 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508016 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.507992 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-client-ca-bundle\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508054 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-cni-bin\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508066 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-run-openvswitch\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508090 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-cni-netd\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508094 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508125 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49hkf\" (UniqueName: \"kubernetes.io/projected/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-kube-api-access-49hkf\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508150 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-metrics-server-audit-profiles\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508195 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-secret-metrics-server-tls\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508178 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508213 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508247 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508238 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ecf8c2f-987e-4441-a30d-2e75f3307aae-metrics-client-ca\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508124 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508283 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-conf-dir\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508310 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508337 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-cni-netd\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508339 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfvxh\" (UniqueName: \"kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-kube-api-access-dfvxh\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508369 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-run-openvswitch\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508381 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-run-multus-certs\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508389 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508407 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-sys\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508340 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-grpc-tls\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508436 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508391 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-conf-dir\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508463 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ecf8c2f-987e-4441-a30d-2e75f3307aae-metrics-client-ca\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508482 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508518 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-run-multus-certs\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508512 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508538 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508576 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-systemd-units\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508597 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-metrics-server-audit-profiles\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508605 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-wtmp\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508653 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508683 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508713 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ecf8c2f-987e-4441-a30d-2e75f3307aae-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508731 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-systemd-units\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508730 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vjcv\" (UniqueName: \"kubernetes.io/projected/a47b32de-3a53-4b34-8874-87049c4d747b-kube-api-access-7vjcv\") pod \"ingress-canary-nwbhl\" (UID: \"a47b32de-3a53-4b34-8874-87049c4d747b\") " pod="openshift-ingress-canary/ingress-canary-nwbhl" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508768 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-etc-openvswitch\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508777 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508795 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508838 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-etc-openvswitch\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508872 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508880 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508899 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508924 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-run\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508945 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-config-out\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508955 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508972 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508979 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-run\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508993 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.508997 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d225fb4-f477-4971-bce1-8aaa24a79be8-system-cni-dir\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509028 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-config-out\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509033 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d225fb4-f477-4971-bce1-8aaa24a79be8-system-cni-dir\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509039 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d225fb4-f477-4971-bce1-8aaa24a79be8-cnibin\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509069 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-run-ovn\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509096 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-var-lib-cni-bin\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509105 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509071 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d225fb4-f477-4971-bce1-8aaa24a79be8-cnibin\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509121 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/afbca029-7d59-4cfb-b08c-353e174b0355-config-volume\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509145 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-var-lib-openvswitch\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509150 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-run-ovn\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509154 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509165 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-var-lib-cni-bin\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509170 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8lqk\" (UniqueName: \"kubernetes.io/projected/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-api-access-k8lqk\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509195 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-var-lib-openvswitch\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509217 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509234 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-textfile\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509250 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-tls\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509280 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509284 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/afbca029-7d59-4cfb-b08c-353e174b0355-config-volume\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509306 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-textfile\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509326 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc-hosts-file\") pod \"node-resolver-5tkc8\" (UID: \"a48715bf-e5aa-4b50-a9a5-4916e6fbefbc\") " pod="openshift-dns/node-resolver-5tkc8" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509354 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-etc-kubernetes\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509375 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-sysconfig\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509417 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-etc-kubernetes\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.509356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509423 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-tls\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509433 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-sysconfig\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509466 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a48715bf-e5aa-4b50-a9a5-4916e6fbefbc-hosts-file\") pod \"node-resolver-5tkc8\" (UID: \"a48715bf-e5aa-4b50-a9a5-4916e6fbefbc\") " pod="openshift-dns/node-resolver-5tkc8" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509491 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-systemd\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509507 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-var-lib-kubelet\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509532 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-root\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509531 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-systemd\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509549 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509580 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnz65\" (UniqueName: \"kubernetes.io/projected/7ecf8c2f-987e-4441-a30d-2e75f3307aae-kube-api-access-tnz65\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509583 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-var-lib-kubelet\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509606 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9372be33-753d-4006-ba16-86fc44bd7c52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509653 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-device-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509678 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-etc-selinux\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509697 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/43f970d6-97ec-4e75-820b-aceefb7c37f5-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4c892\" (UID: \"43f970d6-97ec-4e75-820b-aceefb7c37f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4c892" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509716 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-log-socket\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509743 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-lib-modules\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509764 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-etc-selinux\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509767 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9300fb6-6926-4429-8797-603dfb361818-host\") pod \"node-ca-q28kf\" (UID: \"d9300fb6-6926-4429-8797-603dfb361818\") " pod="openshift-image-registry/node-ca-q28kf" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509785 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11bb6b24-b4e9-4764-a40a-e1daa2c613c2-log-socket\") pod \"ovnkube-node-5wh7f\" (UID: \"11bb6b24-b4e9-4764-a40a-e1daa2c613c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509801 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9300fb6-6926-4429-8797-603dfb361818-host\") pod \"node-ca-q28kf\" (UID: \"d9300fb6-6926-4429-8797-603dfb361818\") " pod="openshift-image-registry/node-ca-q28kf" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509803 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9372be33-753d-4006-ba16-86fc44bd7c52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509743 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509823 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-sys-fs\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509834 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-device-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509850 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-cni-dir\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509873 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-cnibin\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509884 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-sys-fs\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509895 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-host\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509918 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afbca029-7d59-4cfb-b08c-353e174b0355-metrics-tls\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509928 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/43f970d6-97ec-4e75-820b-aceefb7c37f5-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4c892\" (UID: \"43f970d6-97ec-4e75-820b-aceefb7c37f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4c892" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509938 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-cni-dir\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509943 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-modprobe-d\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509948 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-cnibin\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509954 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-host\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509917 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-lib-modules\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.509977 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510034 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqn2p\" (UniqueName: \"kubernetes.io/projected/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-kube-api-access-pqn2p\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510041 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-modprobe-d\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510060 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-registration-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510087 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-system-cni-dir\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510110 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510125 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e6a790-de24-49f1-b507-ed74d1446868-registration-dir\") pod \"aws-ebs-csi-driver-node-sr4f2\" (UID: \"40e6a790-de24-49f1-b507-ed74d1446868\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sr4f2" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510132 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afbca029-7d59-4cfb-b08c-353e174b0355-tmp-dir\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510159 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-system-cni-dir\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510159 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afbca029-7d59-4cfb-b08c-353e174b0355-metrics-tls\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510160 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-socket-dir-parent\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510211 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-multus-socket-dir-parent\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510216 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afbca029-7d59-4cfb-b08c-353e174b0355-tmp-dir\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510238 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-run-netns\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510269 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-audit-log\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510289 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510300 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510317 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510330 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-run-netns\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510338 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510364 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-audit-log\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510371 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-sysctl-conf\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510400 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-sys\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510430 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510486 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510496 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510515 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ecf8c2f-987e-4441-a30d-2e75f3307aae-tmp\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510520 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510528 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-sysctl-conf\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510539 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510544 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9372be33-753d-4006-ba16-86fc44bd7c52-crio-socket\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510577 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-sys\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510576 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d225fb4-f477-4971-bce1-8aaa24a79be8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510609 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ecf8c2f-987e-4441-a30d-2e75f3307aae-tmp\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510640 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-run-k8s-cni-cncf-io\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510648 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d225fb4-f477-4971-bce1-8aaa24a79be8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdll2\" (UID: \"8d225fb4-f477-4971-bce1-8aaa24a79be8\") " pod="openshift-multus/multus-additional-cni-plugins-vdll2" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510686 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-hostroot\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510714 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-kubernetes\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510736 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510743 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-metrics-client-ca\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510762 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-host-run-k8s-cni-cncf-io\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510775 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510782 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3e35ffc8-ef10-4532-af91-38841c3f4581-hostroot\") pod \"multus-gbc4r\" (UID: \"3e35ffc8-ef10-4532-af91-38841c3f4581\") " pod="openshift-multus/multus-gbc4r" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510796 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f379fbd-7dcf-4bff-a4c0-d946e711cf7e-etc-kubernetes\") pod \"tuned-s9kl9\" (UID: \"1f379fbd-7dcf-4bff-a4c0-d946e711cf7e\") " pod="openshift-cluster-node-tuning-operator/tuned-s9kl9" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510804 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510776 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510829 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-web-config\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510853 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510880 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdccp\" (UniqueName: \"kubernetes.io/projected/afbca029-7d59-4cfb-b08c-353e174b0355-kube-api-access-hdccp\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510904 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9372be33-753d-4006-ba16-86fc44bd7c52-data-volume\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510972 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510991 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9372be33-753d-4006-ba16-86fc44bd7c52-data-volume\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510994 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.510976 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-metrics-client-ca\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.511020 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-web-config\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.513414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.511101 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.516995 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.513835 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpg54\" (UniqueName: \"kubernetes.io/projected/9372be33-753d-4006-ba16-86fc44bd7c52-kube-api-access-rpg54\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:56.516995 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.514460 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpp8l\" (UniqueName: \"kubernetes.io/projected/c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8-kube-api-access-dpp8l\") pod \"metrics-server-dcf4d45cc-49kqs\" (UID: \"c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8\") " pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:56.516995 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.514682 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vjcv\" (UniqueName: \"kubernetes.io/projected/a47b32de-3a53-4b34-8874-87049c4d747b-kube-api-access-7vjcv\") pod \"ingress-canary-nwbhl\" (UID: \"a47b32de-3a53-4b34-8874-87049c4d747b\") " pod="openshift-ingress-canary/ingress-canary-nwbhl" Nov 02 17:33:56.516995 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.514976 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49hkf\" (UniqueName: \"kubernetes.io/projected/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-kube-api-access-49hkf\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.516995 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.515164 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwl2c\" (UniqueName: \"kubernetes.io/projected/43f970d6-97ec-4e75-820b-aceefb7c37f5-kube-api-access-wwl2c\") pod \"monitoring-plugin-77f4f76594-4c892\" (UID: \"43f970d6-97ec-4e75-820b-aceefb7c37f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4c892" Nov 02 17:33:56.516995 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.515517 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfvxh\" (UniqueName: \"kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-kube-api-access-dfvxh\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.516995 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.515575 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:56.516995 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.515776 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnz65\" (UniqueName: \"kubernetes.io/projected/7ecf8c2f-987e-4441-a30d-2e75f3307aae-kube-api-access-tnz65\") pod \"thanos-querier-6f98f548cc-srzfq\" (UID: \"7ecf8c2f-987e-4441-a30d-2e75f3307aae\") " pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:33:56.516995 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.515951 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8lqk\" (UniqueName: \"kubernetes.io/projected/065a4c3d-712b-4b65-b18f-0428e7a93672-kube-api-access-k8lqk\") pod \"kube-state-metrics-5dc8d7d89b-zk477\" (UID: \"065a4c3d-712b-4b65-b18f-0428e7a93672\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk477" Nov 02 17:33:56.516995 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.516274 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqn2p\" (UniqueName: \"kubernetes.io/projected/7ae7bace-4c4c-4f60-9863-5b2c29b54b76-kube-api-access-pqn2p\") pod \"openshift-state-metrics-7bd7489957-9vgph\" (UID: \"7ae7bace-4c4c-4f60-9863-5b2c29b54b76\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9vgph" Nov 02 17:33:56.516995 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.516823 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdccp\" (UniqueName: \"kubernetes.io/projected/afbca029-7d59-4cfb-b08c-353e174b0355-kube-api-access-hdccp\") pod \"dns-default-rtxf5\" (UID: \"afbca029-7d59-4cfb-b08c-353e174b0355\") " pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:56.612086 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.612057 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9372be33-753d-4006-ba16-86fc44bd7c52-crio-socket\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:56.612222 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.612106 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-sys\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.612222 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.612129 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-wtmp\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.612222 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.612175 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9372be33-753d-4006-ba16-86fc44bd7c52-crio-socket\") pod \"insights-runtime-extractor-vrm5q\" (UID: \"9372be33-753d-4006-ba16-86fc44bd7c52\") " pod="openshift-insights/insights-runtime-extractor-vrm5q" Nov 02 17:33:56.612222 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.612189 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-node-exporter-wtmp\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.612222 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.612186 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-sys\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.612222 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.612209 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-root\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:56.612453 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:56.612247 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a0753ab-8636-4ce9-9b3e-d89516b73c1c-root\") pod \"node-exporter-8dtcc\" (UID: \"0a0753ab-8636-4ce9-9b3e-d89516b73c1c\") " pod="openshift-monitoring/node-exporter-8dtcc" Nov 02 17:33:57.832812 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:57.832785 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4c892" Nov 02 17:33:58.958717 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:58.958688 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-dcf4d45cc-49kqs" Nov 02 17:33:59.167540 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:59.167515 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:33:59.186804 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:59.186782 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rtxf5" Nov 02 17:33:59.944090 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:33:59.944056 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:34:00.892590 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:00.892559 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6f98f548cc-srzfq" Nov 02 17:34:01.275090 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:01.275057 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:34:01.729742 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:01.729715 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:34:01.729917 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:01.729906 5601 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:34:04.483284 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:34:04.483244 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:34:12.767116 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:12.767083 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cjnbh" Nov 02 17:34:16.588517 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.588484 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:34:16.589041 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.588984 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="prometheus" containerID="cri-o://395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc" gracePeriod=600 Nov 02 17:34:16.589041 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.589003 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="kube-rbac-proxy" containerID="cri-o://a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6" gracePeriod=600 Nov 02 17:34:16.589254 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.589044 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="config-reloader" containerID="cri-o://e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f" gracePeriod=600 Nov 02 17:34:16.589254 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.589017 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831" gracePeriod=600 Nov 02 17:34:16.589254 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.589080 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="thanos-sidecar" containerID="cri-o://25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128" gracePeriod=600 Nov 02 17:34:16.589254 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.589094 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="kube-rbac-proxy-web" containerID="cri-o://df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615" gracePeriod=600 Nov 02 17:34:16.831351 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.831328 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:16.975092 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975067 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-0\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.975280 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975101 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-2\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.975280 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975126 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-thanos-prometheus-http-client-file\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.975280 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975172 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-kubelet-serving-ca-bundle\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.975280 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975204 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-kube-rbac-proxy\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.975280 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975234 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-serving-certs-ca-bundle\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.975280 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975269 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-trusted-ca-bundle\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.975607 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975301 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-tls-assets\" (UniqueName: \"kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-tls-assets\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.975607 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975324 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-web-config\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-web-config\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.975607 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975360 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-1\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.975607 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975402 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-config\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-config\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.975607 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975443 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-metrics-client-certs\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.975607 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975489 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-kube-api-access-dfvxh\" (UniqueName: \"kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-kube-api-access-dfvxh\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.975607 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975516 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-tls\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.975607 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975515 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-2" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:16.975607 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975551 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-grpc-tls\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.975607 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975581 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.976547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975637 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.976547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975671 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-config-out\" (UniqueName: \"kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-config-out\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.976547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975698 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-kubelet-serving-ca-bundle" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:16.976547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975709 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-metrics-client-ca\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.976547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975712 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-serving-certs-ca-bundle" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:16.976547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975748 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-db\") pod \"54ae9769-93f1-4962-b68c-22f5b4658484\" (UID: \"54ae9769-93f1-4962-b68c-22f5b4658484\") " Nov 02 17:34:16.976547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.975924 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-1" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:16.976547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.976089 5601 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:16.976547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.976112 5601 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:16.976547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.976130 5601 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:16.976547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.976152 5601 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:16.976547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.976248 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-trusted-ca-bundle" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:16.977171 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.976672 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-db" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:34:16.977171 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.976830 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-0" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:16.978104 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.978076 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-metrics-client-ca" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:16.979328 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.979299 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-kube-rbac-proxy" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:16.979433 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.979372 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-thanos-prometheus-http-client-file" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:16.979568 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.979537 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-tls-assets" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:34:16.980109 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.980084 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-kube-rbac-proxy-web" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:16.980244 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.980213 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-grpc-tls" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:16.980353 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.980302 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-metrics-client-certs" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:16.980419 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.980390 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-tls" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:16.980477 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.980459 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-kube-api-access-dfvxh" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "kube-api-access-dfvxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:34:16.980528 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.980473 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-config-out" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:34:16.980664 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.980646 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-thanos-sidecar-tls" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:16.981277 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.981259 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-config" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:16.989019 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:16.988998 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-web-config" pod "54ae9769-93f1-4962-b68c-22f5b4658484" (UID: "54ae9769-93f1-4962-b68c-22f5b4658484"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:17.076868 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.076843 5601 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-kube-rbac-proxy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:17.076868 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.076864 5601 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:17.076991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.076877 5601 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-tls-assets\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:17.076991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.076887 5601 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-web-config\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:17.076991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.076895 5601 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-config\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:17.076991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.076904 5601 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-metrics-client-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:17.076991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.076912 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfvxh\" (UniqueName: \"kubernetes.io/projected/54ae9769-93f1-4962-b68c-22f5b4658484-kube-api-access-dfvxh\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:17.076991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.076921 5601 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:17.076991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.076929 5601 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-grpc-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:17.076991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.076938 5601 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:17.076991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.076947 5601 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:17.076991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.076956 5601 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-config-out\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:17.076991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.076964 5601 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-configmap-metrics-client-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:17.076991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.076973 5601 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-db\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:17.076991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.076982 5601 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/54ae9769-93f1-4962-b68c-22f5b4658484-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:17.076991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.076990 5601 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/54ae9769-93f1-4962-b68c-22f5b4658484-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:34:17.471954 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.471923 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5wh7f" Nov 02 17:34:17.566358 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.566321 5601 generic.go:356] "Generic (PLEG): container finished" podID="54ae9769-93f1-4962-b68c-22f5b4658484" containerID="4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831" exitCode=0 Nov 02 17:34:17.566358 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.566346 5601 generic.go:356] "Generic (PLEG): container finished" podID="54ae9769-93f1-4962-b68c-22f5b4658484" containerID="a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6" exitCode=0 Nov 02 17:34:17.566358 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.566353 5601 generic.go:356] "Generic (PLEG): container finished" podID="54ae9769-93f1-4962-b68c-22f5b4658484" containerID="df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615" exitCode=0 Nov 02 17:34:17.566358 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.566360 5601 generic.go:356] "Generic (PLEG): container finished" podID="54ae9769-93f1-4962-b68c-22f5b4658484" containerID="25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128" exitCode=0 Nov 02 17:34:17.566358 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.566367 5601 generic.go:356] "Generic (PLEG): container finished" podID="54ae9769-93f1-4962-b68c-22f5b4658484" containerID="e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f" exitCode=0 Nov 02 17:34:17.566772 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.566374 5601 generic.go:356] "Generic (PLEG): container finished" podID="54ae9769-93f1-4962-b68c-22f5b4658484" containerID="395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc" exitCode=0 Nov 02 17:34:17.566772 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.566393 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ae9769-93f1-4962-b68c-22f5b4658484","Type":"ContainerDied","Data":"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831"} Nov 02 17:34:17.566772 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.566416 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.566772 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.566461 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ae9769-93f1-4962-b68c-22f5b4658484","Type":"ContainerDied","Data":"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6"} Nov 02 17:34:17.566772 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.566489 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ae9769-93f1-4962-b68c-22f5b4658484","Type":"ContainerDied","Data":"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615"} Nov 02 17:34:17.566772 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.566514 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ae9769-93f1-4962-b68c-22f5b4658484","Type":"ContainerDied","Data":"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128"} Nov 02 17:34:17.566772 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.566527 5601 scope.go:122] "RemoveContainer" containerID="4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831" Nov 02 17:34:17.566772 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.566549 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ae9769-93f1-4962-b68c-22f5b4658484","Type":"ContainerDied","Data":"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f"} Nov 02 17:34:17.566772 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.566581 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ae9769-93f1-4962-b68c-22f5b4658484","Type":"ContainerDied","Data":"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc"} Nov 02 17:34:17.566772 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.566609 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ae9769-93f1-4962-b68c-22f5b4658484","Type":"ContainerDied","Data":"1264654e7535055e82ca3e6ab8c24b022d08e7ededcad69edba5051e23b559fc"} Nov 02 17:34:17.574089 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.574069 5601 scope.go:122] "RemoveContainer" containerID="a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6" Nov 02 17:34:17.580883 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.580852 5601 scope.go:122] "RemoveContainer" containerID="df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615" Nov 02 17:34:17.587223 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.587194 5601 scope.go:122] "RemoveContainer" containerID="25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128" Nov 02 17:34:17.588413 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.588395 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:34:17.592930 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.592908 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:34:17.594985 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.594898 5601 scope.go:122] "RemoveContainer" containerID="e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f" Nov 02 17:34:17.601105 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.601082 5601 scope.go:122] "RemoveContainer" containerID="395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc" Nov 02 17:34:17.614868 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.614847 5601 scope.go:122] "RemoveContainer" containerID="7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87" Nov 02 17:34:17.621293 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.621271 5601 scope.go:122] "RemoveContainer" containerID="4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831" Nov 02 17:34:17.621781 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:34:17.621600 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831\": container with ID starting with 4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831 not found: ID does not exist" containerID="4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831" Nov 02 17:34:17.621781 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.621653 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831"} err="failed to get container status \"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831\": rpc error: code = NotFound desc = could not find container \"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831\": container with ID starting with 4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831 not found: ID does not exist" Nov 02 17:34:17.621781 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.621698 5601 scope.go:122] "RemoveContainer" containerID="a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6" Nov 02 17:34:17.622093 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:34:17.622059 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6\": container with ID starting with a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6 not found: ID does not exist" containerID="a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6" Nov 02 17:34:17.622150 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622098 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6"} err="failed to get container status \"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6\": rpc error: code = NotFound desc = could not find container \"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6\": container with ID starting with a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6 not found: ID does not exist" Nov 02 17:34:17.622150 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622127 5601 scope.go:122] "RemoveContainer" containerID="df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615" Nov 02 17:34:17.622419 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:34:17.622400 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615\": container with ID starting with df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615 not found: ID does not exist" containerID="df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615" Nov 02 17:34:17.622515 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622427 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615"} err="failed to get container status \"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615\": rpc error: code = NotFound desc = could not find container \"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615\": container with ID starting with df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615 not found: ID does not exist" Nov 02 17:34:17.622515 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622448 5601 scope.go:122] "RemoveContainer" containerID="25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128" Nov 02 17:34:17.622515 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622491 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:34:17.622747 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:34:17.622727 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128\": container with ID starting with 25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128 not found: ID does not exist" containerID="25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128" Nov 02 17:34:17.622790 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622756 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128"} err="failed to get container status \"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128\": rpc error: code = NotFound desc = could not find container \"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128\": container with ID starting with 25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128 not found: ID does not exist" Nov 02 17:34:17.622790 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622777 5601 scope.go:122] "RemoveContainer" containerID="e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f" Nov 02 17:34:17.622868 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622831 5601 cpu_manager.go:414] "containerMap: removing container" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="kube-rbac-proxy-thanos" Nov 02 17:34:17.622868 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622842 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="kube-rbac-proxy-thanos" Nov 02 17:34:17.622868 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622852 5601 cpu_manager.go:414] "containerMap: removing container" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="prometheus" Nov 02 17:34:17.622868 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622858 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="prometheus" Nov 02 17:34:17.623051 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622871 5601 cpu_manager.go:414] "containerMap: removing container" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="config-reloader" Nov 02 17:34:17.623051 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622880 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="config-reloader" Nov 02 17:34:17.623051 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622893 5601 cpu_manager.go:414] "containerMap: removing container" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="kube-rbac-proxy-web" Nov 02 17:34:17.623051 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622898 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="kube-rbac-proxy-web" Nov 02 17:34:17.623051 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622905 5601 cpu_manager.go:414] "containerMap: removing container" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="thanos-sidecar" Nov 02 17:34:17.623051 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622909 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="thanos-sidecar" Nov 02 17:34:17.623051 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622919 5601 cpu_manager.go:414] "containerMap: removing container" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="kube-rbac-proxy" Nov 02 17:34:17.623051 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622923 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="kube-rbac-proxy" Nov 02 17:34:17.623051 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622937 5601 cpu_manager.go:414] "containerMap: removing container" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="init-config-reloader" Nov 02 17:34:17.623051 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.622971 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="init-config-reloader" Nov 02 17:34:17.623051 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.623042 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="kube-rbac-proxy-thanos" Nov 02 17:34:17.623051 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.623053 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="prometheus" Nov 02 17:34:17.623446 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.623062 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="config-reloader" Nov 02 17:34:17.623446 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:34:17.623063 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f\": container with ID starting with e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f not found: ID does not exist" containerID="e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f" Nov 02 17:34:17.623446 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.623086 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f"} err="failed to get container status \"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f\": rpc error: code = NotFound desc = could not find container \"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f\": container with ID starting with e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f not found: ID does not exist" Nov 02 17:34:17.623446 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.623103 5601 scope.go:122] "RemoveContainer" containerID="395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc" Nov 02 17:34:17.623446 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.623070 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="kube-rbac-proxy-web" Nov 02 17:34:17.623446 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.623151 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="thanos-sidecar" Nov 02 17:34:17.623446 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.623160 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="kube-rbac-proxy" Nov 02 17:34:17.623446 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.623169 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" containerName="init-config-reloader" Nov 02 17:34:17.623446 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:34:17.623339 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc\": container with ID starting with 395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc not found: ID does not exist" containerID="395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc" Nov 02 17:34:17.623446 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.623357 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc"} err="failed to get container status \"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc\": rpc error: code = NotFound desc = could not find container \"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc\": container with ID starting with 395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc not found: ID does not exist" Nov 02 17:34:17.623446 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.623370 5601 scope.go:122] "RemoveContainer" containerID="7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87" Nov 02 17:34:17.623955 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:34:17.623583 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87\": container with ID starting with 7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87 not found: ID does not exist" containerID="7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87" Nov 02 17:34:17.623955 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.623611 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87"} err="failed to get container status \"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87\": rpc error: code = NotFound desc = could not find container \"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87\": container with ID starting with 7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87 not found: ID does not exist" Nov 02 17:34:17.623955 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.623655 5601 scope.go:122] "RemoveContainer" containerID="4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831" Nov 02 17:34:17.623955 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.623873 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831"} err="failed to get container status \"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831\": rpc error: code = NotFound desc = could not find container \"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831\": container with ID starting with 4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831 not found: ID does not exist" Nov 02 17:34:17.623955 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.623887 5601 scope.go:122] "RemoveContainer" containerID="a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6" Nov 02 17:34:17.624114 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.624059 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6"} err="failed to get container status \"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6\": rpc error: code = NotFound desc = could not find container \"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6\": container with ID starting with a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6 not found: ID does not exist" Nov 02 17:34:17.624114 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.624073 5601 scope.go:122] "RemoveContainer" containerID="df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615" Nov 02 17:34:17.624344 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.624326 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615"} err="failed to get container status \"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615\": rpc error: code = NotFound desc = could not find container \"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615\": container with ID starting with df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615 not found: ID does not exist" Nov 02 17:34:17.624383 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.624346 5601 scope.go:122] "RemoveContainer" containerID="25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128" Nov 02 17:34:17.624561 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.624544 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128"} err="failed to get container status \"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128\": rpc error: code = NotFound desc = could not find container \"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128\": container with ID starting with 25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128 not found: ID does not exist" Nov 02 17:34:17.624604 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.624561 5601 scope.go:122] "RemoveContainer" containerID="e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f" Nov 02 17:34:17.624844 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.624825 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f"} err="failed to get container status \"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f\": rpc error: code = NotFound desc = could not find container \"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f\": container with ID starting with e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f not found: ID does not exist" Nov 02 17:34:17.624914 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.624845 5601 scope.go:122] "RemoveContainer" containerID="395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc" Nov 02 17:34:17.625112 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.625095 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc"} err="failed to get container status \"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc\": rpc error: code = NotFound desc = could not find container \"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc\": container with ID starting with 395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc not found: ID does not exist" Nov 02 17:34:17.625180 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.625113 5601 scope.go:122] "RemoveContainer" containerID="7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87" Nov 02 17:34:17.625334 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.625315 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87"} err="failed to get container status \"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87\": rpc error: code = NotFound desc = could not find container \"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87\": container with ID starting with 7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87 not found: ID does not exist" Nov 02 17:34:17.625380 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.625335 5601 scope.go:122] "RemoveContainer" containerID="4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831" Nov 02 17:34:17.625566 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.625549 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831"} err="failed to get container status \"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831\": rpc error: code = NotFound desc = could not find container \"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831\": container with ID starting with 4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831 not found: ID does not exist" Nov 02 17:34:17.625602 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.625565 5601 scope.go:122] "RemoveContainer" containerID="a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6" Nov 02 17:34:17.625752 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.625734 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6"} err="failed to get container status \"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6\": rpc error: code = NotFound desc = could not find container \"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6\": container with ID starting with a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6 not found: ID does not exist" Nov 02 17:34:17.625796 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.625753 5601 scope.go:122] "RemoveContainer" containerID="df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615" Nov 02 17:34:17.625912 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.625898 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615"} err="failed to get container status \"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615\": rpc error: code = NotFound desc = could not find container \"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615\": container with ID starting with df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615 not found: ID does not exist" Nov 02 17:34:17.625958 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.625914 5601 scope.go:122] "RemoveContainer" containerID="25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128" Nov 02 17:34:17.626093 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.626074 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128"} err="failed to get container status \"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128\": rpc error: code = NotFound desc = could not find container \"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128\": container with ID starting with 25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128 not found: ID does not exist" Nov 02 17:34:17.626154 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.626093 5601 scope.go:122] "RemoveContainer" containerID="e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f" Nov 02 17:34:17.626296 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.626280 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f"} err="failed to get container status \"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f\": rpc error: code = NotFound desc = could not find container \"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f\": container with ID starting with e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f not found: ID does not exist" Nov 02 17:34:17.626333 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.626297 5601 scope.go:122] "RemoveContainer" containerID="395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc" Nov 02 17:34:17.626505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.626487 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc"} err="failed to get container status \"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc\": rpc error: code = NotFound desc = could not find container \"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc\": container with ID starting with 395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc not found: ID does not exist" Nov 02 17:34:17.626546 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.626506 5601 scope.go:122] "RemoveContainer" containerID="7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87" Nov 02 17:34:17.626746 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.626730 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87"} err="failed to get container status \"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87\": rpc error: code = NotFound desc = could not find container \"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87\": container with ID starting with 7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87 not found: ID does not exist" Nov 02 17:34:17.626795 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.626746 5601 scope.go:122] "RemoveContainer" containerID="4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831" Nov 02 17:34:17.626962 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.626942 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831"} err="failed to get container status \"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831\": rpc error: code = NotFound desc = could not find container \"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831\": container with ID starting with 4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831 not found: ID does not exist" Nov 02 17:34:17.627042 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.626963 5601 scope.go:122] "RemoveContainer" containerID="a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6" Nov 02 17:34:17.627122 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.627108 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.627227 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.627207 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6"} err="failed to get container status \"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6\": rpc error: code = NotFound desc = could not find container \"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6\": container with ID starting with a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6 not found: ID does not exist" Nov 02 17:34:17.627307 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.627229 5601 scope.go:122] "RemoveContainer" containerID="df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615" Nov 02 17:34:17.627810 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.627787 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615"} err="failed to get container status \"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615\": rpc error: code = NotFound desc = could not find container \"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615\": container with ID starting with df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615 not found: ID does not exist" Nov 02 17:34:17.627810 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.627809 5601 scope.go:122] "RemoveContainer" containerID="25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128" Nov 02 17:34:17.628110 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.628088 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128"} err="failed to get container status \"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128\": rpc error: code = NotFound desc = could not find container \"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128\": container with ID starting with 25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128 not found: ID does not exist" Nov 02 17:34:17.628184 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.628110 5601 scope.go:122] "RemoveContainer" containerID="e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f" Nov 02 17:34:17.628386 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.628368 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f"} err="failed to get container status \"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f\": rpc error: code = NotFound desc = could not find container \"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f\": container with ID starting with e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f not found: ID does not exist" Nov 02 17:34:17.628386 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.628386 5601 scope.go:122] "RemoveContainer" containerID="395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc" Nov 02 17:34:17.628657 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.628635 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc"} err="failed to get container status \"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc\": rpc error: code = NotFound desc = could not find container \"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc\": container with ID starting with 395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc not found: ID does not exist" Nov 02 17:34:17.628742 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.628657 5601 scope.go:122] "RemoveContainer" containerID="7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87" Nov 02 17:34:17.628945 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.628921 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87"} err="failed to get container status \"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87\": rpc error: code = NotFound desc = could not find container \"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87\": container with ID starting with 7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87 not found: ID does not exist" Nov 02 17:34:17.628991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.628948 5601 scope.go:122] "RemoveContainer" containerID="4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831" Nov 02 17:34:17.629185 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.629168 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831"} err="failed to get container status \"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831\": rpc error: code = NotFound desc = could not find container \"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831\": container with ID starting with 4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831 not found: ID does not exist" Nov 02 17:34:17.629227 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.629185 5601 scope.go:122] "RemoveContainer" containerID="a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6" Nov 02 17:34:17.629416 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.629389 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6"} err="failed to get container status \"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6\": rpc error: code = NotFound desc = could not find container \"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6\": container with ID starting with a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6 not found: ID does not exist" Nov 02 17:34:17.629457 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.629417 5601 scope.go:122] "RemoveContainer" containerID="df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615" Nov 02 17:34:17.629674 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.629654 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615"} err="failed to get container status \"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615\": rpc error: code = NotFound desc = could not find container \"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615\": container with ID starting with df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615 not found: ID does not exist" Nov 02 17:34:17.629727 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.629675 5601 scope.go:122] "RemoveContainer" containerID="25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128" Nov 02 17:34:17.629894 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.629877 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128"} err="failed to get container status \"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128\": rpc error: code = NotFound desc = could not find container \"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128\": container with ID starting with 25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128 not found: ID does not exist" Nov 02 17:34:17.629942 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.629894 5601 scope.go:122] "RemoveContainer" containerID="e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f" Nov 02 17:34:17.630117 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.630097 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f"} err="failed to get container status \"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f\": rpc error: code = NotFound desc = could not find container \"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f\": container with ID starting with e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f not found: ID does not exist" Nov 02 17:34:17.630154 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.630119 5601 scope.go:122] "RemoveContainer" containerID="395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc" Nov 02 17:34:17.630377 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.630362 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc"} err="failed to get container status \"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc\": rpc error: code = NotFound desc = could not find container \"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc\": container with ID starting with 395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc not found: ID does not exist" Nov 02 17:34:17.630417 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.630377 5601 scope.go:122] "RemoveContainer" containerID="7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87" Nov 02 17:34:17.630606 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.630592 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87"} err="failed to get container status \"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87\": rpc error: code = NotFound desc = could not find container \"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87\": container with ID starting with 7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87 not found: ID does not exist" Nov 02 17:34:17.630666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.630606 5601 scope.go:122] "RemoveContainer" containerID="4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831" Nov 02 17:34:17.630834 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.630818 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831"} err="failed to get container status \"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831\": rpc error: code = NotFound desc = could not find container \"4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831\": container with ID starting with 4d4501fda01eeda2449c85bf6ee54cb572069c1fe412b5e350f17c96d9718831 not found: ID does not exist" Nov 02 17:34:17.630873 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.630834 5601 scope.go:122] "RemoveContainer" containerID="a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6" Nov 02 17:34:17.631050 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.631034 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6"} err="failed to get container status \"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6\": rpc error: code = NotFound desc = could not find container \"a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6\": container with ID starting with a103332dd06691caa86c03a884c213e852b594d50f1889757c8d9e51d9173ea6 not found: ID does not exist" Nov 02 17:34:17.631050 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.631048 5601 scope.go:122] "RemoveContainer" containerID="df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615" Nov 02 17:34:17.631298 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.631279 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615"} err="failed to get container status \"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615\": rpc error: code = NotFound desc = could not find container \"df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615\": container with ID starting with df412ad663d1cf8b5880bf66c0da25dcdf9c085a6e4f9f3c6546dfc7c77c5615 not found: ID does not exist" Nov 02 17:34:17.631346 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.631299 5601 scope.go:122] "RemoveContainer" containerID="25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128" Nov 02 17:34:17.631531 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.631510 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128"} err="failed to get container status \"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128\": rpc error: code = NotFound desc = could not find container \"25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128\": container with ID starting with 25f35112058d0d7619e97a6b31551d64eebcd47613542c00f58d4b4c6fc0b128 not found: ID does not exist" Nov 02 17:34:17.631531 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.631529 5601 scope.go:122] "RemoveContainer" containerID="e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f" Nov 02 17:34:17.631749 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.631732 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f"} err="failed to get container status \"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f\": rpc error: code = NotFound desc = could not find container \"e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f\": container with ID starting with e35905005cbc8236ac7fdf80c8898abe1a9ff8176d195c0e2b5efb252732401f not found: ID does not exist" Nov 02 17:34:17.631820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.631750 5601 scope.go:122] "RemoveContainer" containerID="395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc" Nov 02 17:34:17.632089 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.632068 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc"} err="failed to get container status \"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc\": rpc error: code = NotFound desc = could not find container \"395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc\": container with ID starting with 395a167eee179fd3e2189fe510538b8c5f5f398faf036d1d19a7a25561f089dc not found: ID does not exist" Nov 02 17:34:17.632089 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.632088 5601 scope.go:122] "RemoveContainer" containerID="7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87" Nov 02 17:34:17.632300 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.632286 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 17:34:17.632365 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.632286 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 17:34:17.632365 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.632286 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 17:34:17.632365 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.632340 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87"} err="failed to get container status \"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87\": rpc error: code = NotFound desc = could not find container \"7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87\": container with ID starting with 7530c612d9a67493316d3423fff680aa48b40105774dfa8d61036dd883b43f87 not found: ID does not exist" Nov 02 17:34:17.632513 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.632386 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 17:34:17.632513 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.632387 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 17:34:17.632513 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.632455 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 17:34:17.632513 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.632487 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cs83l4avv8s1v\"" Nov 02 17:34:17.632513 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.632491 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 17:34:17.632839 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.632585 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 17:34:17.632839 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.632651 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wrf7q\"" Nov 02 17:34:17.632839 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.632697 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 17:34:17.632839 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.632720 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 17:34:17.632839 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.632792 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 17:34:17.633016 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.632841 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 17:34:17.635887 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.635869 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 17:34:17.635969 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.635956 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 17:34:17.650277 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.650255 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:34:17.783593 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.783525 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45570963-375c-49e4-bda8-01a095e015f7-config-out\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.783593 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.783559 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.783593 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.783576 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.783807 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.783596 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.783807 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.783657 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l92c4\" (UniqueName: \"kubernetes.io/projected/45570963-375c-49e4-bda8-01a095e015f7-kube-api-access-l92c4\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.783807 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.783691 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.783807 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.783724 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.783807 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.783749 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.783807 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.783784 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.783984 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.783871 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-config\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.783984 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.783904 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.783984 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.783922 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45570963-375c-49e4-bda8-01a095e015f7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.783984 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.783938 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.784097 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.783996 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.784097 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.784012 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.784097 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.784028 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-web-config\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.784097 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.784060 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.784214 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.784111 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.784214 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.784132 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45570963-375c-49e4-bda8-01a095e015f7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.784214 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.784151 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.884760 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.884727 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-config\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.884760 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.884761 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.884978 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.884780 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45570963-375c-49e4-bda8-01a095e015f7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.884978 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.884909 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.884978 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.884957 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.885137 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.884982 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.885137 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.885003 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-web-config\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.885137 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.885025 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.885137 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.885065 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.885137 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.885099 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45570963-375c-49e4-bda8-01a095e015f7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.885137 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.885123 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.885435 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.885164 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45570963-375c-49e4-bda8-01a095e015f7-config-out\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.885435 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.885176 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45570963-375c-49e4-bda8-01a095e015f7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.885435 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.885189 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.885435 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.885211 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.885435 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.885209 5601 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 17:34:17.885435 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.885247 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.885435 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.885270 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l92c4\" (UniqueName: \"kubernetes.io/projected/45570963-375c-49e4-bda8-01a095e015f7-kube-api-access-l92c4\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.885435 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.885300 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.885435 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.885326 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.885435 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.885351 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.885435 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.885376 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.886010 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.885455 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.886010 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.885676 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.886234 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.886202 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.886305 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.886259 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.887070 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.886733 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.887070 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.886999 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.888975 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.888948 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.889696 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.889655 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.889696 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.889660 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45570963-375c-49e4-bda8-01a095e015f7-config-out\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.889838 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.889822 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.890405 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.889923 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45570963-375c-49e4-bda8-01a095e015f7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.890405 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.890348 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.890405 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.890358 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45570963-375c-49e4-bda8-01a095e015f7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.890405 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.890352 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.890586 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.890417 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.890586 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.890476 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.890885 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.890869 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-config\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.891479 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.891461 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45570963-375c-49e4-bda8-01a095e015f7-web-config\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.895305 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.895282 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l92c4\" (UniqueName: \"kubernetes.io/projected/45570963-375c-49e4-bda8-01a095e015f7-kube-api-access-l92c4\") pod \"prometheus-k8s-0\" (UID: \"45570963-375c-49e4-bda8-01a095e015f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:17.947105 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:17.947081 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:18.230818 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:18.230785 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:34:18.233955 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:34:18.233928 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45570963_375c_49e4_bda8_01a095e015f7.slice/crio-e68fc147abe39f1085a32f3ed4c3033c15a7e4511d1cf6d435911b09f0fe2779 WatchSource:0}: Error finding container e68fc147abe39f1085a32f3ed4c3033c15a7e4511d1cf6d435911b09f0fe2779: Status 404 returned error can't find the container with id e68fc147abe39f1085a32f3ed4c3033c15a7e4511d1cf6d435911b09f0fe2779 Nov 02 17:34:18.571262 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:18.571182 5601 generic.go:356] "Generic (PLEG): container finished" podID="45570963-375c-49e4-bda8-01a095e015f7" containerID="b045086c6dc776e5f27aef83421b50779cb4dfb71153110b5953240282d09c23" exitCode=0 Nov 02 17:34:18.571412 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:18.571264 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45570963-375c-49e4-bda8-01a095e015f7","Type":"ContainerDied","Data":"b045086c6dc776e5f27aef83421b50779cb4dfb71153110b5953240282d09c23"} Nov 02 17:34:18.571412 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:18.571346 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45570963-375c-49e4-bda8-01a095e015f7","Type":"ContainerStarted","Data":"e68fc147abe39f1085a32f3ed4c3033c15a7e4511d1cf6d435911b09f0fe2779"} Nov 02 17:34:19.423283 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:19.423250 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54ae9769-93f1-4962-b68c-22f5b4658484" path="/var/lib/kubelet/pods/54ae9769-93f1-4962-b68c-22f5b4658484/volumes" Nov 02 17:34:19.578863 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:19.578828 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45570963-375c-49e4-bda8-01a095e015f7","Type":"ContainerStarted","Data":"df29cd94aa2a21b1dd2b73b16b71d93bc4660027355b58b13d5bd74b7ac51da8"} Nov 02 17:34:19.579006 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:19.578906 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45570963-375c-49e4-bda8-01a095e015f7","Type":"ContainerStarted","Data":"d9c6b902b4938e6c0c94e4b318e880a0115062e8a4371c975b9c2c331c9436dc"} Nov 02 17:34:19.579006 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:19.578948 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45570963-375c-49e4-bda8-01a095e015f7","Type":"ContainerStarted","Data":"1356934428f2fd74961fce7314bbf10c1b5f8f48888638bcce1525684a74100e"} Nov 02 17:34:19.579006 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:19.578985 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45570963-375c-49e4-bda8-01a095e015f7","Type":"ContainerStarted","Data":"f0070d05e21c9cae07021a83c2b684336e13702b3c5a01ee2d831ea4f0ec3e9d"} Nov 02 17:34:19.579108 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:19.579018 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45570963-375c-49e4-bda8-01a095e015f7","Type":"ContainerStarted","Data":"dc6fbe65284918b5a4200136f94f5be743ca7fdeaa51f8e405b2c350d443188f"} Nov 02 17:34:19.579108 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:19.579054 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45570963-375c-49e4-bda8-01a095e015f7","Type":"ContainerStarted","Data":"72187b7242ebb8fef4b47a4b8ff1443e2dbf67083703d835bc3881bcaec93e20"} Nov 02 17:34:19.613977 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:19.613920 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.597823799 podStartE2EDuration="2.61390858s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.084781ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:19.61176497 +0000 UTC m=+24.302818982" watchObservedRunningTime="2026-11-02 17:34:19.61390858 +0000 UTC m=+24.304962589" Nov 02 17:34:22.948202 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:22.948160 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:29.947165 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:29.947129 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-424d5" Nov 02 17:34:34.482134 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:34:34.482088 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:34:34.741291 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:34.741212 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:34.756423 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:34.756402 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:34:35.654566 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:34:35.654537 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:35:04.483050 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:35:04.483017 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:35:34.483383 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:35:34.483335 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:35:45.930322 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:45.930233 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs"] Nov 02 17:35:45.932786 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:45.932768 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs" Nov 02 17:35:45.937930 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:45.937906 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-vcjd6\"" Nov 02 17:35:45.939175 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:45.939154 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 17:35:45.939249 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:45.939158 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 17:35:45.944000 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:45.943981 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs"] Nov 02 17:35:45.977485 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:45.977457 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3891638-e257-4ed3-958a-e00f9da23fdc-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs\" (UID: \"b3891638-e257-4ed3-958a-e00f9da23fdc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs" Nov 02 17:35:45.977644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:45.977490 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3891638-e257-4ed3-958a-e00f9da23fdc-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs\" (UID: \"b3891638-e257-4ed3-958a-e00f9da23fdc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs" Nov 02 17:35:45.977644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:45.977519 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq5zw\" (UniqueName: \"kubernetes.io/projected/b3891638-e257-4ed3-958a-e00f9da23fdc-kube-api-access-nq5zw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs\" (UID: \"b3891638-e257-4ed3-958a-e00f9da23fdc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs" Nov 02 17:35:46.078637 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:46.078589 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3891638-e257-4ed3-958a-e00f9da23fdc-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs\" (UID: \"b3891638-e257-4ed3-958a-e00f9da23fdc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs" Nov 02 17:35:46.078637 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:46.078643 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3891638-e257-4ed3-958a-e00f9da23fdc-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs\" (UID: \"b3891638-e257-4ed3-958a-e00f9da23fdc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs" Nov 02 17:35:46.078820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:46.078676 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq5zw\" (UniqueName: \"kubernetes.io/projected/b3891638-e257-4ed3-958a-e00f9da23fdc-kube-api-access-nq5zw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs\" (UID: \"b3891638-e257-4ed3-958a-e00f9da23fdc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs" Nov 02 17:35:46.078996 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:46.078978 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3891638-e257-4ed3-958a-e00f9da23fdc-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs\" (UID: \"b3891638-e257-4ed3-958a-e00f9da23fdc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs" Nov 02 17:35:46.079032 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:46.078995 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3891638-e257-4ed3-958a-e00f9da23fdc-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs\" (UID: \"b3891638-e257-4ed3-958a-e00f9da23fdc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs" Nov 02 17:35:46.087304 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:46.087276 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq5zw\" (UniqueName: \"kubernetes.io/projected/b3891638-e257-4ed3-958a-e00f9da23fdc-kube-api-access-nq5zw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs\" (UID: \"b3891638-e257-4ed3-958a-e00f9da23fdc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs" Nov 02 17:35:46.247045 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:46.246970 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs" Nov 02 17:35:46.526767 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:46.526585 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs"] Nov 02 17:35:46.529261 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:35:46.529232 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3891638_e257_4ed3_958a_e00f9da23fdc.slice/crio-2f258be4d077a0d2f049d42fb8501f172bef23f1fe1cb28bf436e9870f579d56 WatchSource:0}: Error finding container 2f258be4d077a0d2f049d42fb8501f172bef23f1fe1cb28bf436e9870f579d56: Status 404 returned error can't find the container with id 2f258be4d077a0d2f049d42fb8501f172bef23f1fe1cb28bf436e9870f579d56 Nov 02 17:35:46.531166 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:46.531146 5601 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:35:46.881337 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:46.881260 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs" event={"ID":"b3891638-e257-4ed3-958a-e00f9da23fdc","Type":"ContainerStarted","Data":"2f258be4d077a0d2f049d42fb8501f172bef23f1fe1cb28bf436e9870f579d56"} Nov 02 17:35:47.906660 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:47.906608 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g"] Nov 02 17:35:47.909207 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:47.909184 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g" Nov 02 17:35:47.914267 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:47.914246 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-7t6kh\"" Nov 02 17:35:47.925256 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:47.925233 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g"] Nov 02 17:35:47.993891 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:47.993863 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkw9g\" (UniqueName: \"kubernetes.io/projected/2368ef14-831b-4d32-baa5-cd4b52a6e767-kube-api-access-hkw9g\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g\" (UID: \"2368ef14-831b-4d32-baa5-cd4b52a6e767\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g" Nov 02 17:35:47.994042 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:47.993907 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2368ef14-831b-4d32-baa5-cd4b52a6e767-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g\" (UID: \"2368ef14-831b-4d32-baa5-cd4b52a6e767\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g" Nov 02 17:35:47.994096 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:47.994030 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2368ef14-831b-4d32-baa5-cd4b52a6e767-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g\" (UID: \"2368ef14-831b-4d32-baa5-cd4b52a6e767\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g" Nov 02 17:35:48.095146 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:48.095109 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkw9g\" (UniqueName: \"kubernetes.io/projected/2368ef14-831b-4d32-baa5-cd4b52a6e767-kube-api-access-hkw9g\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g\" (UID: \"2368ef14-831b-4d32-baa5-cd4b52a6e767\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g" Nov 02 17:35:48.095146 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:48.095150 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2368ef14-831b-4d32-baa5-cd4b52a6e767-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g\" (UID: \"2368ef14-831b-4d32-baa5-cd4b52a6e767\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g" Nov 02 17:35:48.095394 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:48.095281 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2368ef14-831b-4d32-baa5-cd4b52a6e767-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g\" (UID: \"2368ef14-831b-4d32-baa5-cd4b52a6e767\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g" Nov 02 17:35:48.095668 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:48.095645 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2368ef14-831b-4d32-baa5-cd4b52a6e767-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g\" (UID: \"2368ef14-831b-4d32-baa5-cd4b52a6e767\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g" Nov 02 17:35:48.095763 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:48.095747 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2368ef14-831b-4d32-baa5-cd4b52a6e767-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g\" (UID: \"2368ef14-831b-4d32-baa5-cd4b52a6e767\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g" Nov 02 17:35:48.104542 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:48.104520 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkw9g\" (UniqueName: \"kubernetes.io/projected/2368ef14-831b-4d32-baa5-cd4b52a6e767-kube-api-access-hkw9g\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g\" (UID: \"2368ef14-831b-4d32-baa5-cd4b52a6e767\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g" Nov 02 17:35:48.223877 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:48.223844 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g" Nov 02 17:35:48.508638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:48.508547 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g"] Nov 02 17:35:48.512552 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:35:48.512511 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2368ef14_831b_4d32_baa5_cd4b52a6e767.slice/crio-5b7d2baa9e3d31d133c7bd9a54e98e00824dfb31a9365e6950ee88e0996601f5 WatchSource:0}: Error finding container 5b7d2baa9e3d31d133c7bd9a54e98e00824dfb31a9365e6950ee88e0996601f5: Status 404 returned error can't find the container with id 5b7d2baa9e3d31d133c7bd9a54e98e00824dfb31a9365e6950ee88e0996601f5 Nov 02 17:35:48.890331 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:48.890243 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g" event={"ID":"2368ef14-831b-4d32-baa5-cd4b52a6e767","Type":"ContainerStarted","Data":"5b7d2baa9e3d31d133c7bd9a54e98e00824dfb31a9365e6950ee88e0996601f5"} Nov 02 17:35:49.646219 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.645346 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b"] Nov 02 17:35:49.649703 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.649677 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2"] Nov 02 17:35:49.649848 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.649826 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b" Nov 02 17:35:49.652144 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.652124 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" Nov 02 17:35:49.658436 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.658405 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-6cp72\"" Nov 02 17:35:49.660297 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.660106 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b"] Nov 02 17:35:49.660297 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.660268 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-hwrh9\"" Nov 02 17:35:49.660930 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.660907 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2"] Nov 02 17:35:49.709266 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.709228 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7590cd68-b16b-4dd9-9730-385b6d25b0ab-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b\" (UID: \"7590cd68-b16b-4dd9-9730-385b6d25b0ab\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b" Nov 02 17:35:49.709457 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.709278 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2\" (UID: \"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" Nov 02 17:35:49.709457 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.709344 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2\" (UID: \"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" Nov 02 17:35:49.709457 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.709374 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk8gf\" (UniqueName: \"kubernetes.io/projected/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-kube-api-access-jk8gf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2\" (UID: \"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" Nov 02 17:35:49.709457 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.709421 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldrcn\" (UniqueName: \"kubernetes.io/projected/7590cd68-b16b-4dd9-9730-385b6d25b0ab-kube-api-access-ldrcn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b\" (UID: \"7590cd68-b16b-4dd9-9730-385b6d25b0ab\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b" Nov 02 17:35:49.709748 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.709464 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7590cd68-b16b-4dd9-9730-385b6d25b0ab-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b\" (UID: \"7590cd68-b16b-4dd9-9730-385b6d25b0ab\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b" Nov 02 17:35:49.711121 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.711087 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6"] Nov 02 17:35:49.714346 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.714324 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6" Nov 02 17:35:49.724115 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.723560 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6"] Nov 02 17:35:49.725730 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.725705 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-54zj2\"" Nov 02 17:35:49.810682 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.810639 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2\" (UID: \"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" Nov 02 17:35:49.810682 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.810681 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8"] Nov 02 17:35:49.811386 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.811344 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2\" (UID: \"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" Nov 02 17:35:49.811488 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.810689 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk8gf\" (UniqueName: \"kubernetes.io/projected/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-kube-api-access-jk8gf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2\" (UID: \"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" Nov 02 17:35:49.811652 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.811536 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dzmg\" (UniqueName: \"kubernetes.io/projected/6ddddfdc-ba16-4779-a051-85d73b50c415-kube-api-access-4dzmg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6\" (UID: \"6ddddfdc-ba16-4779-a051-85d73b50c415\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6" Nov 02 17:35:49.811652 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.811575 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldrcn\" (UniqueName: \"kubernetes.io/projected/7590cd68-b16b-4dd9-9730-385b6d25b0ab-kube-api-access-ldrcn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b\" (UID: \"7590cd68-b16b-4dd9-9730-385b6d25b0ab\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b" Nov 02 17:35:49.811652 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.811642 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7590cd68-b16b-4dd9-9730-385b6d25b0ab-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b\" (UID: \"7590cd68-b16b-4dd9-9730-385b6d25b0ab\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b" Nov 02 17:35:49.811817 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.811669 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ddddfdc-ba16-4779-a051-85d73b50c415-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6\" (UID: \"6ddddfdc-ba16-4779-a051-85d73b50c415\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6" Nov 02 17:35:49.811817 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.811713 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7590cd68-b16b-4dd9-9730-385b6d25b0ab-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b\" (UID: \"7590cd68-b16b-4dd9-9730-385b6d25b0ab\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b" Nov 02 17:35:49.811817 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.811734 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ddddfdc-ba16-4779-a051-85d73b50c415-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6\" (UID: \"6ddddfdc-ba16-4779-a051-85d73b50c415\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6" Nov 02 17:35:49.811817 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.811762 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2\" (UID: \"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" Nov 02 17:35:49.812078 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.812059 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2\" (UID: \"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" Nov 02 17:35:49.812580 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.812561 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7590cd68-b16b-4dd9-9730-385b6d25b0ab-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b\" (UID: \"7590cd68-b16b-4dd9-9730-385b6d25b0ab\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b" Nov 02 17:35:49.812828 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.812807 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7590cd68-b16b-4dd9-9730-385b6d25b0ab-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b\" (UID: \"7590cd68-b16b-4dd9-9730-385b6d25b0ab\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b" Nov 02 17:35:49.814322 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.813943 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8" Nov 02 17:35:49.824993 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.823725 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-5cz8l\"" Nov 02 17:35:49.824993 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.824172 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk8gf\" (UniqueName: \"kubernetes.io/projected/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-kube-api-access-jk8gf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2\" (UID: \"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" Nov 02 17:35:49.824993 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.824902 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8"] Nov 02 17:35:49.826869 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.826787 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldrcn\" (UniqueName: \"kubernetes.io/projected/7590cd68-b16b-4dd9-9730-385b6d25b0ab-kube-api-access-ldrcn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b\" (UID: \"7590cd68-b16b-4dd9-9730-385b6d25b0ab\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b" Nov 02 17:35:49.912637 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.912524 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ddddfdc-ba16-4779-a051-85d73b50c415-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6\" (UID: \"6ddddfdc-ba16-4779-a051-85d73b50c415\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6" Nov 02 17:35:49.912637 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.912583 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ddddfdc-ba16-4779-a051-85d73b50c415-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6\" (UID: \"6ddddfdc-ba16-4779-a051-85d73b50c415\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6" Nov 02 17:35:49.912637 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.912606 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8\" (UID: \"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8" Nov 02 17:35:49.912930 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.912650 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb98k\" (UniqueName: \"kubernetes.io/projected/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-kube-api-access-zb98k\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8\" (UID: \"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8" Nov 02 17:35:49.912930 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.912693 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8\" (UID: \"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8" Nov 02 17:35:49.912930 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.912727 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dzmg\" (UniqueName: \"kubernetes.io/projected/6ddddfdc-ba16-4779-a051-85d73b50c415-kube-api-access-4dzmg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6\" (UID: \"6ddddfdc-ba16-4779-a051-85d73b50c415\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6" Nov 02 17:35:49.913376 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.913352 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ddddfdc-ba16-4779-a051-85d73b50c415-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6\" (UID: \"6ddddfdc-ba16-4779-a051-85d73b50c415\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6" Nov 02 17:35:49.913648 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.913604 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ddddfdc-ba16-4779-a051-85d73b50c415-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6\" (UID: \"6ddddfdc-ba16-4779-a051-85d73b50c415\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6" Nov 02 17:35:49.922839 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.922807 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dzmg\" (UniqueName: \"kubernetes.io/projected/6ddddfdc-ba16-4779-a051-85d73b50c415-kube-api-access-4dzmg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6\" (UID: \"6ddddfdc-ba16-4779-a051-85d73b50c415\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6" Nov 02 17:35:49.978033 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.977992 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b" Nov 02 17:35:49.985497 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:49.985254 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" Nov 02 17:35:50.013551 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:50.013517 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb98k\" (UniqueName: \"kubernetes.io/projected/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-kube-api-access-zb98k\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8\" (UID: \"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8" Nov 02 17:35:50.013686 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:50.013605 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8\" (UID: \"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8" Nov 02 17:35:50.013751 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:50.013715 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8\" (UID: \"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8" Nov 02 17:35:50.014126 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:50.014102 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8\" (UID: \"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8" Nov 02 17:35:50.014303 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:50.014251 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8\" (UID: \"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8" Nov 02 17:35:50.025219 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:50.025187 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb98k\" (UniqueName: \"kubernetes.io/projected/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-kube-api-access-zb98k\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8\" (UID: \"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8" Nov 02 17:35:50.032208 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:50.031930 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6" Nov 02 17:35:50.149383 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:50.149184 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8" Nov 02 17:35:50.267547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:50.267515 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b"] Nov 02 17:35:50.271908 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:35:50.271875 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7590cd68_b16b_4dd9_9730_385b6d25b0ab.slice/crio-1cd49fa5e7316fafc82a4390d6c7cc0d7abb151f4f84a5ab94cde326308141e0 WatchSource:0}: Error finding container 1cd49fa5e7316fafc82a4390d6c7cc0d7abb151f4f84a5ab94cde326308141e0: Status 404 returned error can't find the container with id 1cd49fa5e7316fafc82a4390d6c7cc0d7abb151f4f84a5ab94cde326308141e0 Nov 02 17:35:50.294434 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:50.294401 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2"] Nov 02 17:35:50.298176 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:35:50.298148 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d5a59b_0a0b_4fb4_ac8f_36ffa6388b9f.slice/crio-ff8fac8ccfa0f678b37402493f062d77a9091b6b86c2e8d3bd501e6d3627ce66 WatchSource:0}: Error finding container ff8fac8ccfa0f678b37402493f062d77a9091b6b86c2e8d3bd501e6d3627ce66: Status 404 returned error can't find the container with id ff8fac8ccfa0f678b37402493f062d77a9091b6b86c2e8d3bd501e6d3627ce66 Nov 02 17:35:50.353084 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:50.353052 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6"] Nov 02 17:35:50.356282 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:35:50.356245 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ddddfdc_ba16_4779_a051_85d73b50c415.slice/crio-1afec7c31ab3caabe30a49affc4253196881867458f7ec59ead95ed63263359c WatchSource:0}: Error finding container 1afec7c31ab3caabe30a49affc4253196881867458f7ec59ead95ed63263359c: Status 404 returned error can't find the container with id 1afec7c31ab3caabe30a49affc4253196881867458f7ec59ead95ed63263359c Nov 02 17:35:50.440807 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:50.440772 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8"] Nov 02 17:35:50.444424 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:35:50.444393 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4c55b0_0320_4e4c_a349_7dda4b2d2c58.slice/crio-1ed5467b270d6162cb50267db1d18fadf989a5680f97f4dbdfe8b72d0bedd30d WatchSource:0}: Error finding container 1ed5467b270d6162cb50267db1d18fadf989a5680f97f4dbdfe8b72d0bedd30d: Status 404 returned error can't find the container with id 1ed5467b270d6162cb50267db1d18fadf989a5680f97f4dbdfe8b72d0bedd30d Nov 02 17:35:50.901719 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:50.901580 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8" event={"ID":"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58","Type":"ContainerStarted","Data":"1ed5467b270d6162cb50267db1d18fadf989a5680f97f4dbdfe8b72d0bedd30d"} Nov 02 17:35:50.905770 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:50.905699 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6" event={"ID":"6ddddfdc-ba16-4779-a051-85d73b50c415","Type":"ContainerStarted","Data":"1afec7c31ab3caabe30a49affc4253196881867458f7ec59ead95ed63263359c"} Nov 02 17:35:50.910018 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:50.909954 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b" event={"ID":"7590cd68-b16b-4dd9-9730-385b6d25b0ab","Type":"ContainerStarted","Data":"1cd49fa5e7316fafc82a4390d6c7cc0d7abb151f4f84a5ab94cde326308141e0"} Nov 02 17:35:50.911893 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:50.911837 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" event={"ID":"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f","Type":"ContainerStarted","Data":"ff8fac8ccfa0f678b37402493f062d77a9091b6b86c2e8d3bd501e6d3627ce66"} Nov 02 17:35:51.591518 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.591455 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5"] Nov 02 17:35:51.595547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.595526 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.609154 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.608853 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 17:35:51.609154 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.608939 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 17:35:51.609154 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.609026 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 17:35:51.609154 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.609116 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qbv6v\"" Nov 02 17:35:51.610717 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.610694 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5"] Nov 02 17:35:51.631754 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.631724 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f70366-70f1-475a-8e45-eb6da6b4828e-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.631902 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.631780 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrqv9\" (UniqueName: \"kubernetes.io/projected/63f70366-70f1-475a-8e45-eb6da6b4828e-kube-api-access-xrqv9\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.631902 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.631821 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f70366-70f1-475a-8e45-eb6da6b4828e-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.631902 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.631858 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f70366-70f1-475a-8e45-eb6da6b4828e-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.632076 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.631900 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f70366-70f1-475a-8e45-eb6da6b4828e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.632076 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.631950 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f70366-70f1-475a-8e45-eb6da6b4828e-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.632076 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.631978 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f70366-70f1-475a-8e45-eb6da6b4828e-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.632076 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.632022 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f70366-70f1-475a-8e45-eb6da6b4828e-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.632076 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.632055 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f70366-70f1-475a-8e45-eb6da6b4828e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.733814 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.732965 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f70366-70f1-475a-8e45-eb6da6b4828e-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.733814 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.733007 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f70366-70f1-475a-8e45-eb6da6b4828e-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.733814 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.733046 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f70366-70f1-475a-8e45-eb6da6b4828e-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.733814 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.733070 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f70366-70f1-475a-8e45-eb6da6b4828e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.733814 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.733138 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f70366-70f1-475a-8e45-eb6da6b4828e-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.733814 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.733178 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrqv9\" (UniqueName: \"kubernetes.io/projected/63f70366-70f1-475a-8e45-eb6da6b4828e-kube-api-access-xrqv9\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.733814 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.733211 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f70366-70f1-475a-8e45-eb6da6b4828e-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.733814 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.733246 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f70366-70f1-475a-8e45-eb6da6b4828e-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.733814 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.733288 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f70366-70f1-475a-8e45-eb6da6b4828e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.733814 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.733369 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f70366-70f1-475a-8e45-eb6da6b4828e-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.733814 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.733552 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f70366-70f1-475a-8e45-eb6da6b4828e-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.733814 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.733793 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f70366-70f1-475a-8e45-eb6da6b4828e-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.734538 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.734060 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f70366-70f1-475a-8e45-eb6da6b4828e-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.735517 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.734357 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f70366-70f1-475a-8e45-eb6da6b4828e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.736611 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.736584 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f70366-70f1-475a-8e45-eb6da6b4828e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.738524 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.738500 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f70366-70f1-475a-8e45-eb6da6b4828e-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.765708 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.765556 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrqv9\" (UniqueName: \"kubernetes.io/projected/63f70366-70f1-475a-8e45-eb6da6b4828e-kube-api-access-xrqv9\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.775676 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.775649 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f70366-70f1-475a-8e45-eb6da6b4828e-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-wtks5\" (UID: \"63f70366-70f1-475a-8e45-eb6da6b4828e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:51.914739 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:51.914219 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:35:52.215753 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:52.215588 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5"] Nov 02 17:35:52.220600 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:35:52.220566 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f70366_70f1_475a_8e45_eb6da6b4828e.slice/crio-38b96e569e100ca44f65efce5630dc01a99a07efac7081c472368b1d2bf1fd05 WatchSource:0}: Error finding container 38b96e569e100ca44f65efce5630dc01a99a07efac7081c472368b1d2bf1fd05: Status 404 returned error can't find the container with id 38b96e569e100ca44f65efce5630dc01a99a07efac7081c472368b1d2bf1fd05 Nov 02 17:35:52.937664 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:52.937571 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" event={"ID":"63f70366-70f1-475a-8e45-eb6da6b4828e","Type":"ContainerStarted","Data":"38b96e569e100ca44f65efce5630dc01a99a07efac7081c472368b1d2bf1fd05"} Nov 02 17:35:55.975671 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:55.975634 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs" event={"ID":"b3891638-e257-4ed3-958a-e00f9da23fdc","Type":"ContainerStarted","Data":"71c1f7cc9ea928eabad6df5bd95feb197050f8f631f6f23a51639b151dc44ca0"} Nov 02 17:35:56.996488 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:56.995498 5601 generic.go:356] "Generic (PLEG): container finished" podID="b3891638-e257-4ed3-958a-e00f9da23fdc" containerID="71c1f7cc9ea928eabad6df5bd95feb197050f8f631f6f23a51639b151dc44ca0" exitCode=0 Nov 02 17:35:56.996488 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:56.995528 5601 generic.go:356] "Generic (PLEG): container finished" podID="b3891638-e257-4ed3-958a-e00f9da23fdc" containerID="6c49ed5f832dccc9982035795db4d745a0334927ce98b92c3b45e5aae98be627" exitCode=0 Nov 02 17:35:56.996488 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:56.995636 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs" event={"ID":"b3891638-e257-4ed3-958a-e00f9da23fdc","Type":"ContainerDied","Data":"71c1f7cc9ea928eabad6df5bd95feb197050f8f631f6f23a51639b151dc44ca0"} Nov 02 17:35:56.996488 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:56.995675 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs" event={"ID":"b3891638-e257-4ed3-958a-e00f9da23fdc","Type":"ContainerDied","Data":"6c49ed5f832dccc9982035795db4d745a0334927ce98b92c3b45e5aae98be627"} Nov 02 17:35:59.015985 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:59.015956 5601 generic.go:356] "Generic (PLEG): container finished" podID="6ddddfdc-ba16-4779-a051-85d73b50c415" containerID="b3ef09bddb5e35a67c42872604c9dcdf72fb7f7a2c1287259288f175159ddc6c" exitCode=0 Nov 02 17:35:59.016367 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:59.016089 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6" event={"ID":"6ddddfdc-ba16-4779-a051-85d73b50c415","Type":"ContainerDied","Data":"b3ef09bddb5e35a67c42872604c9dcdf72fb7f7a2c1287259288f175159ddc6c"} Nov 02 17:35:59.689048 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:59.689008 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:35:59.689184 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:59.689100 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:35:59.689184 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:35:59.689153 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:36:00.022812 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:00.022776 5601 generic.go:356] "Generic (PLEG): container finished" podID="2368ef14-831b-4d32-baa5-cd4b52a6e767" containerID="058251a94f44f0e2c71c18dfdcb3532f11e526d8d135b79946960414be49441b" exitCode=0 Nov 02 17:36:00.023251 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:00.022870 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g" event={"ID":"2368ef14-831b-4d32-baa5-cd4b52a6e767","Type":"ContainerDied","Data":"058251a94f44f0e2c71c18dfdcb3532f11e526d8d135b79946960414be49441b"} Nov 02 17:36:00.025005 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:00.024981 5601 generic.go:356] "Generic (PLEG): container finished" podID="7590cd68-b16b-4dd9-9730-385b6d25b0ab" containerID="5761054e1c3176f11e7528f7dfcf7c697f75a2fa39c4d13afef3a45f124c7f40" exitCode=0 Nov 02 17:36:00.025103 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:00.025036 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b" event={"ID":"7590cd68-b16b-4dd9-9730-385b6d25b0ab","Type":"ContainerDied","Data":"5761054e1c3176f11e7528f7dfcf7c697f75a2fa39c4d13afef3a45f124c7f40"} Nov 02 17:36:00.027471 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:00.027327 5601 generic.go:356] "Generic (PLEG): container finished" podID="73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f" containerID="64a272589fdbc66dd3044421975b46b076cf03e61f6dd4b503a7ab7102c11fd1" exitCode=0 Nov 02 17:36:00.027471 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:00.027451 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" event={"ID":"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f","Type":"ContainerDied","Data":"64a272589fdbc66dd3044421975b46b076cf03e61f6dd4b503a7ab7102c11fd1"} Nov 02 17:36:00.029593 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:00.029572 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" event={"ID":"63f70366-70f1-475a-8e45-eb6da6b4828e","Type":"ContainerStarted","Data":"ef9d8fd509c7278e026a2048271a935ccca8262d5ce15ad0c2bc2896aa6eaa6d"} Nov 02 17:36:00.031208 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:00.031186 5601 generic.go:356] "Generic (PLEG): container finished" podID="bb4c55b0-0320-4e4c-a349-7dda4b2d2c58" containerID="11cc89ab9b554d946bd86b796895742bde251e037d8c37941cc2e3df4d255ce2" exitCode=0 Nov 02 17:36:00.031317 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:00.031214 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8" event={"ID":"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58","Type":"ContainerDied","Data":"11cc89ab9b554d946bd86b796895742bde251e037d8c37941cc2e3df4d255ce2"} Nov 02 17:36:00.151564 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:00.150540 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" podStartSLOduration=1.6864942040000002 podStartE2EDuration="9.150527278s" totalImagesPullingTime="7.464033074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:00.149890421 +0000 UTC m=+124.840944434" watchObservedRunningTime="2026-11-02 17:36:00.150527278 +0000 UTC m=+124.841581289" Nov 02 17:36:00.915633 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:00.915543 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:36:00.921179 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:00.921155 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:36:01.037178 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:01.037142 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b" event={"ID":"7590cd68-b16b-4dd9-9730-385b6d25b0ab","Type":"ContainerStarted","Data":"4c19355af65ea1d2805e7939262b25498a2275e949993dc42aecde9f92a651b0"} Nov 02 17:36:01.039565 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:01.039539 5601 generic.go:356] "Generic (PLEG): container finished" podID="6ddddfdc-ba16-4779-a051-85d73b50c415" containerID="eeb03421b067f3904d205a733d521763ec96309a5b46e04434575c79bf382639" exitCode=0 Nov 02 17:36:01.039670 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:01.039597 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6" event={"ID":"6ddddfdc-ba16-4779-a051-85d73b50c415","Type":"ContainerDied","Data":"eeb03421b067f3904d205a733d521763ec96309a5b46e04434575c79bf382639"} Nov 02 17:36:01.040056 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:01.040016 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:36:01.041437 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:01.041420 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:36:02.048880 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:02.048843 5601 generic.go:356] "Generic (PLEG): container finished" podID="7590cd68-b16b-4dd9-9730-385b6d25b0ab" containerID="4c19355af65ea1d2805e7939262b25498a2275e949993dc42aecde9f92a651b0" exitCode=0 Nov 02 17:36:02.049473 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:02.048947 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b" event={"ID":"7590cd68-b16b-4dd9-9730-385b6d25b0ab","Type":"ContainerDied","Data":"4c19355af65ea1d2805e7939262b25498a2275e949993dc42aecde9f92a651b0"} Nov 02 17:36:02.052678 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:02.052652 5601 generic.go:356] "Generic (PLEG): container finished" podID="73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f" containerID="5dc8f45c012ea32f03b24e9df60945a8a0f0588f4aba5afe26180c7e41d97bc3" exitCode=0 Nov 02 17:36:02.052825 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:02.052808 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" event={"ID":"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f","Type":"ContainerDied","Data":"5dc8f45c012ea32f03b24e9df60945a8a0f0588f4aba5afe26180c7e41d97bc3"} Nov 02 17:36:02.056389 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:02.056367 5601 generic.go:356] "Generic (PLEG): container finished" podID="bb4c55b0-0320-4e4c-a349-7dda4b2d2c58" containerID="1588f6509f0caee3b86fca72f6b199244d8e1f066d27d9271c3e3918bc2f34f7" exitCode=0 Nov 02 17:36:02.057371 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:02.057350 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8" event={"ID":"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58","Type":"ContainerDied","Data":"1588f6509f0caee3b86fca72f6b199244d8e1f066d27d9271c3e3918bc2f34f7"} Nov 02 17:36:04.074430 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:04.074392 5601 generic.go:356] "Generic (PLEG): container finished" podID="2368ef14-831b-4d32-baa5-cd4b52a6e767" containerID="e55cb75cfc4aaf3cd376892b4c6130817c0b44439c21fb166d5770bd9ad82ebc" exitCode=0 Nov 02 17:36:04.074881 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:04.074499 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g" event={"ID":"2368ef14-831b-4d32-baa5-cd4b52a6e767","Type":"ContainerDied","Data":"e55cb75cfc4aaf3cd376892b4c6130817c0b44439c21fb166d5770bd9ad82ebc"} Nov 02 17:36:04.514127 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:36:04.513957 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:36:07.096481 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:07.096411 5601 generic.go:356] "Generic (PLEG): container finished" podID="b3891638-e257-4ed3-958a-e00f9da23fdc" containerID="f744111a3daa839440ce77b4897967480158f3e98204047fc8c4f6f26b23975b" exitCode=0 Nov 02 17:36:07.096964 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:07.096540 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs" event={"ID":"b3891638-e257-4ed3-958a-e00f9da23fdc","Type":"ContainerDied","Data":"f744111a3daa839440ce77b4897967480158f3e98204047fc8c4f6f26b23975b"} Nov 02 17:36:08.111449 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:08.111407 5601 generic.go:356] "Generic (PLEG): container finished" podID="6ddddfdc-ba16-4779-a051-85d73b50c415" containerID="4690bc1f4009e9e02387f0bc7cc39e9026c3cb7f591a6d652efa47dcb69b1722" exitCode=0 Nov 02 17:36:08.111906 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:08.111536 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6" event={"ID":"6ddddfdc-ba16-4779-a051-85d73b50c415","Type":"ContainerDied","Data":"4690bc1f4009e9e02387f0bc7cc39e9026c3cb7f591a6d652efa47dcb69b1722"} Nov 02 17:36:08.282086 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:08.281860 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs" Nov 02 17:36:08.411209 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:08.411172 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3891638-e257-4ed3-958a-e00f9da23fdc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3891638-e257-4ed3-958a-e00f9da23fdc-bundle\") pod \"b3891638-e257-4ed3-958a-e00f9da23fdc\" (UID: \"b3891638-e257-4ed3-958a-e00f9da23fdc\") " Nov 02 17:36:08.411385 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:08.411240 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3891638-e257-4ed3-958a-e00f9da23fdc-kube-api-access-nq5zw\" (UniqueName: \"kubernetes.io/projected/b3891638-e257-4ed3-958a-e00f9da23fdc-kube-api-access-nq5zw\") pod \"b3891638-e257-4ed3-958a-e00f9da23fdc\" (UID: \"b3891638-e257-4ed3-958a-e00f9da23fdc\") " Nov 02 17:36:08.411385 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:08.411274 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3891638-e257-4ed3-958a-e00f9da23fdc-util\" (UniqueName: \"kubernetes.io/empty-dir/b3891638-e257-4ed3-958a-e00f9da23fdc-util\") pod \"b3891638-e257-4ed3-958a-e00f9da23fdc\" (UID: \"b3891638-e257-4ed3-958a-e00f9da23fdc\") " Nov 02 17:36:08.412208 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:08.412165 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3891638-e257-4ed3-958a-e00f9da23fdc-bundle" pod "b3891638-e257-4ed3-958a-e00f9da23fdc" (UID: "b3891638-e257-4ed3-958a-e00f9da23fdc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:36:08.414799 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:08.414759 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3891638-e257-4ed3-958a-e00f9da23fdc-kube-api-access-nq5zw" pod "b3891638-e257-4ed3-958a-e00f9da23fdc" (UID: "b3891638-e257-4ed3-958a-e00f9da23fdc"). InnerVolumeSpecName "kube-api-access-nq5zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:36:08.417765 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:08.417735 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3891638-e257-4ed3-958a-e00f9da23fdc-util" pod "b3891638-e257-4ed3-958a-e00f9da23fdc" (UID: "b3891638-e257-4ed3-958a-e00f9da23fdc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:36:08.512705 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:08.512612 5601 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3891638-e257-4ed3-958a-e00f9da23fdc-bundle\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:08.512705 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:08.512667 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq5zw\" (UniqueName: \"kubernetes.io/projected/b3891638-e257-4ed3-958a-e00f9da23fdc-kube-api-access-nq5zw\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:08.512705 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:08.512682 5601 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3891638-e257-4ed3-958a-e00f9da23fdc-util\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:09.119681 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:09.119645 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" event={"ID":"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f","Type":"ContainerStarted","Data":"2d8fdb688caf75a06e81eaec23b8c932440eab33b581e4279bfd5184b3bf0235"} Nov 02 17:36:09.122408 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:09.122385 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs" event={"ID":"b3891638-e257-4ed3-958a-e00f9da23fdc","Type":"ContainerDied","Data":"2f258be4d077a0d2f049d42fb8501f172bef23f1fe1cb28bf436e9870f579d56"} Nov 02 17:36:09.122408 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:09.122396 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71gbtvs" Nov 02 17:36:09.122588 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:09.122425 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f258be4d077a0d2f049d42fb8501f172bef23f1fe1cb28bf436e9870f579d56" Nov 02 17:36:09.142954 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:09.142904 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" podStartSLOduration=3.293827051 podStartE2EDuration="20.142884938s" totalImagesPullingTime="16.849057887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:09.14105343 +0000 UTC m=+133.832107458" watchObservedRunningTime="2026-11-02 17:36:09.142884938 +0000 UTC m=+133.833938952" Nov 02 17:36:09.289442 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:09.289404 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6" Nov 02 17:36:09.420295 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:09.420267 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ddddfdc-ba16-4779-a051-85d73b50c415-kube-api-access-4dzmg\" (UniqueName: \"kubernetes.io/projected/6ddddfdc-ba16-4779-a051-85d73b50c415-kube-api-access-4dzmg\") pod \"6ddddfdc-ba16-4779-a051-85d73b50c415\" (UID: \"6ddddfdc-ba16-4779-a051-85d73b50c415\") " Nov 02 17:36:09.420295 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:09.420321 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ddddfdc-ba16-4779-a051-85d73b50c415-util\" (UniqueName: \"kubernetes.io/empty-dir/6ddddfdc-ba16-4779-a051-85d73b50c415-util\") pod \"6ddddfdc-ba16-4779-a051-85d73b50c415\" (UID: \"6ddddfdc-ba16-4779-a051-85d73b50c415\") " Nov 02 17:36:09.420510 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:09.420393 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ddddfdc-ba16-4779-a051-85d73b50c415-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ddddfdc-ba16-4779-a051-85d73b50c415-bundle\") pod \"6ddddfdc-ba16-4779-a051-85d73b50c415\" (UID: \"6ddddfdc-ba16-4779-a051-85d73b50c415\") " Nov 02 17:36:09.421237 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:09.421194 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ddddfdc-ba16-4779-a051-85d73b50c415-bundle" pod "6ddddfdc-ba16-4779-a051-85d73b50c415" (UID: "6ddddfdc-ba16-4779-a051-85d73b50c415"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:36:09.422015 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:09.421972 5601 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ddddfdc-ba16-4779-a051-85d73b50c415-bundle\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:09.422684 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:09.422651 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ddddfdc-ba16-4779-a051-85d73b50c415-kube-api-access-4dzmg" pod "6ddddfdc-ba16-4779-a051-85d73b50c415" (UID: "6ddddfdc-ba16-4779-a051-85d73b50c415"). InnerVolumeSpecName "kube-api-access-4dzmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:36:09.426017 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:09.425994 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ddddfdc-ba16-4779-a051-85d73b50c415-util" pod "6ddddfdc-ba16-4779-a051-85d73b50c415" (UID: "6ddddfdc-ba16-4779-a051-85d73b50c415"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:36:09.523155 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:09.523085 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dzmg\" (UniqueName: \"kubernetes.io/projected/6ddddfdc-ba16-4779-a051-85d73b50c415-kube-api-access-4dzmg\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:09.523155 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:09.523113 5601 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ddddfdc-ba16-4779-a051-85d73b50c415-util\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:10.128081 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:10.128051 5601 generic.go:356] "Generic (PLEG): container finished" podID="bb4c55b0-0320-4e4c-a349-7dda4b2d2c58" containerID="138e5073ac355a2aef29ef6a3354fa9d21584e5cf148c2cd9ee43eac9f58a32c" exitCode=0 Nov 02 17:36:10.128519 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:10.128130 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8" event={"ID":"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58","Type":"ContainerDied","Data":"138e5073ac355a2aef29ef6a3354fa9d21584e5cf148c2cd9ee43eac9f58a32c"} Nov 02 17:36:10.129869 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:10.129852 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6" Nov 02 17:36:10.129869 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:10.129859 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmqc6" event={"ID":"6ddddfdc-ba16-4779-a051-85d73b50c415","Type":"ContainerDied","Data":"1afec7c31ab3caabe30a49affc4253196881867458f7ec59ead95ed63263359c"} Nov 02 17:36:10.130047 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:10.129893 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1afec7c31ab3caabe30a49affc4253196881867458f7ec59ead95ed63263359c" Nov 02 17:36:10.131967 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:10.131944 5601 generic.go:356] "Generic (PLEG): container finished" podID="7590cd68-b16b-4dd9-9730-385b6d25b0ab" containerID="de6713ff1eed7ec9343926026037950c158da4021254a3f4029333a2f5755a86" exitCode=0 Nov 02 17:36:10.132108 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:10.132027 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b" event={"ID":"7590cd68-b16b-4dd9-9730-385b6d25b0ab","Type":"ContainerDied","Data":"de6713ff1eed7ec9343926026037950c158da4021254a3f4029333a2f5755a86"} Nov 02 17:36:10.134015 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:10.133992 5601 generic.go:356] "Generic (PLEG): container finished" podID="73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f" containerID="2d8fdb688caf75a06e81eaec23b8c932440eab33b581e4279bfd5184b3bf0235" exitCode=0 Nov 02 17:36:10.134130 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:10.134042 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" event={"ID":"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f","Type":"ContainerDied","Data":"2d8fdb688caf75a06e81eaec23b8c932440eab33b581e4279bfd5184b3bf0235"} Nov 02 17:36:11.140265 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.140233 5601 generic.go:356] "Generic (PLEG): container finished" podID="2368ef14-831b-4d32-baa5-cd4b52a6e767" containerID="8a6e5e156410118f0f3753c8d077649f8580eaaeaa587cd29e2472b29c8d34a4" exitCode=0 Nov 02 17:36:11.140752 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.140320 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g" event={"ID":"2368ef14-831b-4d32-baa5-cd4b52a6e767","Type":"ContainerDied","Data":"8a6e5e156410118f0f3753c8d077649f8580eaaeaa587cd29e2472b29c8d34a4"} Nov 02 17:36:11.270664 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.270310 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" Nov 02 17:36:11.301263 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.301241 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b" Nov 02 17:36:11.320412 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.320388 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8" Nov 02 17:36:11.338627 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.338585 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-bundle\") pod \"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f\" (UID: \"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f\") " Nov 02 17:36:11.338762 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.338719 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-kube-api-access-jk8gf\" (UniqueName: \"kubernetes.io/projected/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-kube-api-access-jk8gf\") pod \"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f\" (UID: \"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f\") " Nov 02 17:36:11.338762 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.338744 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-util\" (UniqueName: \"kubernetes.io/empty-dir/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-util\") pod \"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f\" (UID: \"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f\") " Nov 02 17:36:11.339285 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.339257 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-bundle" pod "73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f" (UID: "73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:36:11.340792 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.340770 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-kube-api-access-jk8gf" pod "73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f" (UID: "73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f"). InnerVolumeSpecName "kube-api-access-jk8gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:36:11.345472 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.345448 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-util" pod "73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f" (UID: "73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:36:11.439421 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.439400 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7590cd68-b16b-4dd9-9730-385b6d25b0ab-kube-api-access-ldrcn\" (UniqueName: \"kubernetes.io/projected/7590cd68-b16b-4dd9-9730-385b6d25b0ab-kube-api-access-ldrcn\") pod \"7590cd68-b16b-4dd9-9730-385b6d25b0ab\" (UID: \"7590cd68-b16b-4dd9-9730-385b6d25b0ab\") " Nov 02 17:36:11.439542 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.439435 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-kube-api-access-zb98k\" (UniqueName: \"kubernetes.io/projected/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-kube-api-access-zb98k\") pod \"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58\" (UID: \"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58\") " Nov 02 17:36:11.439542 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.439456 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7590cd68-b16b-4dd9-9730-385b6d25b0ab-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7590cd68-b16b-4dd9-9730-385b6d25b0ab-bundle\") pod \"7590cd68-b16b-4dd9-9730-385b6d25b0ab\" (UID: \"7590cd68-b16b-4dd9-9730-385b6d25b0ab\") " Nov 02 17:36:11.439542 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.439483 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-bundle\") pod \"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58\" (UID: \"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58\") " Nov 02 17:36:11.439542 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.439528 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-util\" (UniqueName: \"kubernetes.io/empty-dir/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-util\") pod \"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58\" (UID: \"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58\") " Nov 02 17:36:11.439769 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.439574 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7590cd68-b16b-4dd9-9730-385b6d25b0ab-util\" (UniqueName: \"kubernetes.io/empty-dir/7590cd68-b16b-4dd9-9730-385b6d25b0ab-util\") pod \"7590cd68-b16b-4dd9-9730-385b6d25b0ab\" (UID: \"7590cd68-b16b-4dd9-9730-385b6d25b0ab\") " Nov 02 17:36:11.439823 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.439806 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk8gf\" (UniqueName: \"kubernetes.io/projected/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-kube-api-access-jk8gf\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:11.439888 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.439828 5601 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-util\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:11.439888 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.439842 5601 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f-bundle\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:11.440092 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.440039 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-bundle" pod "bb4c55b0-0320-4e4c-a349-7dda4b2d2c58" (UID: "bb4c55b0-0320-4e4c-a349-7dda4b2d2c58"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:36:11.440339 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.440309 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7590cd68-b16b-4dd9-9730-385b6d25b0ab-bundle" pod "7590cd68-b16b-4dd9-9730-385b6d25b0ab" (UID: "7590cd68-b16b-4dd9-9730-385b6d25b0ab"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:36:11.445178 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.445150 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-util" pod "bb4c55b0-0320-4e4c-a349-7dda4b2d2c58" (UID: "bb4c55b0-0320-4e4c-a349-7dda4b2d2c58"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:36:11.445669 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.445645 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7590cd68-b16b-4dd9-9730-385b6d25b0ab-util" pod "7590cd68-b16b-4dd9-9730-385b6d25b0ab" (UID: "7590cd68-b16b-4dd9-9730-385b6d25b0ab"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:36:11.453282 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.452881 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-kube-api-access-zb98k" pod "bb4c55b0-0320-4e4c-a349-7dda4b2d2c58" (UID: "bb4c55b0-0320-4e4c-a349-7dda4b2d2c58"). InnerVolumeSpecName "kube-api-access-zb98k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:36:11.464925 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.464903 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7590cd68-b16b-4dd9-9730-385b6d25b0ab-kube-api-access-ldrcn" pod "7590cd68-b16b-4dd9-9730-385b6d25b0ab" (UID: "7590cd68-b16b-4dd9-9730-385b6d25b0ab"). InnerVolumeSpecName "kube-api-access-ldrcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:36:11.540509 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.540479 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldrcn\" (UniqueName: \"kubernetes.io/projected/7590cd68-b16b-4dd9-9730-385b6d25b0ab-kube-api-access-ldrcn\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:11.540509 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.540503 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb98k\" (UniqueName: \"kubernetes.io/projected/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-kube-api-access-zb98k\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:11.540509 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.540515 5601 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7590cd68-b16b-4dd9-9730-385b6d25b0ab-bundle\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:11.540795 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.540524 5601 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-bundle\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:11.540795 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.540532 5601 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb4c55b0-0320-4e4c-a349-7dda4b2d2c58-util\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:11.540795 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:11.540539 5601 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7590cd68-b16b-4dd9-9730-385b6d25b0ab-util\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:12.146254 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:12.146217 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b" event={"ID":"7590cd68-b16b-4dd9-9730-385b6d25b0ab","Type":"ContainerDied","Data":"1cd49fa5e7316fafc82a4390d6c7cc0d7abb151f4f84a5ab94cde326308141e0"} Nov 02 17:36:12.146703 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:12.146301 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cd49fa5e7316fafc82a4390d6c7cc0d7abb151f4f84a5ab94cde326308141e0" Nov 02 17:36:12.146703 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:12.146233 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahsh6b" Nov 02 17:36:12.148175 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:12.148153 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" Nov 02 17:36:12.148301 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:12.148167 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cclkw2" event={"ID":"73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f","Type":"ContainerDied","Data":"ff8fac8ccfa0f678b37402493f062d77a9091b6b86c2e8d3bd501e6d3627ce66"} Nov 02 17:36:12.148301 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:12.148211 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff8fac8ccfa0f678b37402493f062d77a9091b6b86c2e8d3bd501e6d3627ce66" Nov 02 17:36:12.150065 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:12.150018 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8" event={"ID":"bb4c55b0-0320-4e4c-a349-7dda4b2d2c58","Type":"ContainerDied","Data":"1ed5467b270d6162cb50267db1d18fadf989a5680f97f4dbdfe8b72d0bedd30d"} Nov 02 17:36:12.150172 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:12.150068 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ed5467b270d6162cb50267db1d18fadf989a5680f97f4dbdfe8b72d0bedd30d" Nov 02 17:36:12.150231 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:12.150215 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64r7f8" Nov 02 17:36:12.276400 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:12.276378 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g" Nov 02 17:36:12.347789 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:12.347763 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2368ef14-831b-4d32-baa5-cd4b52a6e767-kube-api-access-hkw9g\" (UniqueName: \"kubernetes.io/projected/2368ef14-831b-4d32-baa5-cd4b52a6e767-kube-api-access-hkw9g\") pod \"2368ef14-831b-4d32-baa5-cd4b52a6e767\" (UID: \"2368ef14-831b-4d32-baa5-cd4b52a6e767\") " Nov 02 17:36:12.347909 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:12.347809 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2368ef14-831b-4d32-baa5-cd4b52a6e767-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2368ef14-831b-4d32-baa5-cd4b52a6e767-bundle\") pod \"2368ef14-831b-4d32-baa5-cd4b52a6e767\" (UID: \"2368ef14-831b-4d32-baa5-cd4b52a6e767\") " Nov 02 17:36:12.347909 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:12.347884 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2368ef14-831b-4d32-baa5-cd4b52a6e767-util\" (UniqueName: \"kubernetes.io/empty-dir/2368ef14-831b-4d32-baa5-cd4b52a6e767-util\") pod \"2368ef14-831b-4d32-baa5-cd4b52a6e767\" (UID: \"2368ef14-831b-4d32-baa5-cd4b52a6e767\") " Nov 02 17:36:12.348231 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:12.348207 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2368ef14-831b-4d32-baa5-cd4b52a6e767-bundle" pod "2368ef14-831b-4d32-baa5-cd4b52a6e767" (UID: "2368ef14-831b-4d32-baa5-cd4b52a6e767"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:36:12.349855 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:12.349823 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2368ef14-831b-4d32-baa5-cd4b52a6e767-kube-api-access-hkw9g" pod "2368ef14-831b-4d32-baa5-cd4b52a6e767" (UID: "2368ef14-831b-4d32-baa5-cd4b52a6e767"). InnerVolumeSpecName "kube-api-access-hkw9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:36:12.353204 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:12.353180 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2368ef14-831b-4d32-baa5-cd4b52a6e767-util" pod "2368ef14-831b-4d32-baa5-cd4b52a6e767" (UID: "2368ef14-831b-4d32-baa5-cd4b52a6e767"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:36:12.448568 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:12.448545 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkw9g\" (UniqueName: \"kubernetes.io/projected/2368ef14-831b-4d32-baa5-cd4b52a6e767-kube-api-access-hkw9g\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:12.448710 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:12.448589 5601 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2368ef14-831b-4d32-baa5-cd4b52a6e767-bundle\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:12.448710 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:12.448599 5601 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2368ef14-831b-4d32-baa5-cd4b52a6e767-util\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:36:13.155175 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:13.155147 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g" Nov 02 17:36:13.155175 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:13.155162 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lrb9g" event={"ID":"2368ef14-831b-4d32-baa5-cd4b52a6e767","Type":"ContainerDied","Data":"5b7d2baa9e3d31d133c7bd9a54e98e00824dfb31a9365e6950ee88e0996601f5"} Nov 02 17:36:13.155593 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:13.155201 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b7d2baa9e3d31d133c7bd9a54e98e00824dfb31a9365e6950ee88e0996601f5" Nov 02 17:36:16.135861 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.135828 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-csqq9"] Nov 02 17:36:16.136227 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136136 5601 cpu_manager.go:414] "containerMap: removing container" podUID="73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f" containerName="extract" Nov 02 17:36:16.136227 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136148 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f" containerName="extract" Nov 02 17:36:16.136227 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136182 5601 cpu_manager.go:414] "containerMap: removing container" podUID="7590cd68-b16b-4dd9-9730-385b6d25b0ab" containerName="extract" Nov 02 17:36:16.136227 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136187 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7590cd68-b16b-4dd9-9730-385b6d25b0ab" containerName="extract" Nov 02 17:36:16.136227 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136195 5601 cpu_manager.go:414] "containerMap: removing container" podUID="2368ef14-831b-4d32-baa5-cd4b52a6e767" containerName="extract" Nov 02 17:36:16.136227 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136200 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2368ef14-831b-4d32-baa5-cd4b52a6e767" containerName="extract" Nov 02 17:36:16.136227 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136209 5601 cpu_manager.go:414] "containerMap: removing container" podUID="bb4c55b0-0320-4e4c-a349-7dda4b2d2c58" containerName="util" Nov 02 17:36:16.136227 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136213 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb4c55b0-0320-4e4c-a349-7dda4b2d2c58" containerName="util" Nov 02 17:36:16.136227 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136220 5601 cpu_manager.go:414] "containerMap: removing container" podUID="bb4c55b0-0320-4e4c-a349-7dda4b2d2c58" containerName="pull" Nov 02 17:36:16.136227 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136224 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb4c55b0-0320-4e4c-a349-7dda4b2d2c58" containerName="pull" Nov 02 17:36:16.136227 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136234 5601 cpu_manager.go:414] "containerMap: removing container" podUID="b3891638-e257-4ed3-958a-e00f9da23fdc" containerName="extract" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136239 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3891638-e257-4ed3-958a-e00f9da23fdc" containerName="extract" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136245 5601 cpu_manager.go:414] "containerMap: removing container" podUID="bb4c55b0-0320-4e4c-a349-7dda4b2d2c58" containerName="extract" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136249 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb4c55b0-0320-4e4c-a349-7dda4b2d2c58" containerName="extract" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136257 5601 cpu_manager.go:414] "containerMap: removing container" podUID="b3891638-e257-4ed3-958a-e00f9da23fdc" containerName="util" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136261 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3891638-e257-4ed3-958a-e00f9da23fdc" containerName="util" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136267 5601 cpu_manager.go:414] "containerMap: removing container" podUID="b3891638-e257-4ed3-958a-e00f9da23fdc" containerName="pull" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136271 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3891638-e257-4ed3-958a-e00f9da23fdc" containerName="pull" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136277 5601 cpu_manager.go:414] "containerMap: removing container" podUID="73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f" containerName="util" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136281 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f" containerName="util" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136287 5601 cpu_manager.go:414] "containerMap: removing container" podUID="6ddddfdc-ba16-4779-a051-85d73b50c415" containerName="pull" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136291 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ddddfdc-ba16-4779-a051-85d73b50c415" containerName="pull" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136296 5601 cpu_manager.go:414] "containerMap: removing container" podUID="7590cd68-b16b-4dd9-9730-385b6d25b0ab" containerName="pull" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136300 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7590cd68-b16b-4dd9-9730-385b6d25b0ab" containerName="pull" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136312 5601 cpu_manager.go:414] "containerMap: removing container" podUID="6ddddfdc-ba16-4779-a051-85d73b50c415" containerName="util" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136316 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ddddfdc-ba16-4779-a051-85d73b50c415" containerName="util" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136321 5601 cpu_manager.go:414] "containerMap: removing container" podUID="7590cd68-b16b-4dd9-9730-385b6d25b0ab" containerName="util" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136326 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7590cd68-b16b-4dd9-9730-385b6d25b0ab" containerName="util" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136331 5601 cpu_manager.go:414] "containerMap: removing container" podUID="73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f" containerName="pull" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136335 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f" containerName="pull" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136343 5601 cpu_manager.go:414] "containerMap: removing container" podUID="2368ef14-831b-4d32-baa5-cd4b52a6e767" containerName="util" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136347 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2368ef14-831b-4d32-baa5-cd4b52a6e767" containerName="util" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136354 5601 cpu_manager.go:414] "containerMap: removing container" podUID="6ddddfdc-ba16-4779-a051-85d73b50c415" containerName="extract" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136359 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ddddfdc-ba16-4779-a051-85d73b50c415" containerName="extract" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136365 5601 cpu_manager.go:414] "containerMap: removing container" podUID="2368ef14-831b-4d32-baa5-cd4b52a6e767" containerName="pull" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136370 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2368ef14-831b-4d32-baa5-cd4b52a6e767" containerName="pull" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136415 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ddddfdc-ba16-4779-a051-85d73b50c415" containerName="extract" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136423 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d5a59b-0a0b-4fb4-ac8f-36ffa6388b9f" containerName="extract" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136429 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="7590cd68-b16b-4dd9-9730-385b6d25b0ab" containerName="extract" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136434 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="2368ef14-831b-4d32-baa5-cd4b52a6e767" containerName="extract" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136441 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3891638-e257-4ed3-958a-e00f9da23fdc" containerName="extract" Nov 02 17:36:16.136559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.136446 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb4c55b0-0320-4e4c-a349-7dda4b2d2c58" containerName="extract" Nov 02 17:36:16.183359 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.183320 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-csqq9"] Nov 02 17:36:16.183533 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.183501 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-csqq9" Nov 02 17:36:16.187917 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.187886 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-wm2cc\"" Nov 02 17:36:16.189038 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.189006 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:36:16.189038 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.189012 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 02 17:36:16.279491 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.279457 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4249b8fd-e594-496c-82f5-1c37fa2b6b1d-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-csqq9\" (UID: \"4249b8fd-e594-496c-82f5-1c37fa2b6b1d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-csqq9" Nov 02 17:36:16.279491 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.279492 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nlhp\" (UniqueName: \"kubernetes.io/projected/4249b8fd-e594-496c-82f5-1c37fa2b6b1d-kube-api-access-5nlhp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-csqq9\" (UID: \"4249b8fd-e594-496c-82f5-1c37fa2b6b1d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-csqq9" Nov 02 17:36:16.379962 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.379925 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4249b8fd-e594-496c-82f5-1c37fa2b6b1d-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-csqq9\" (UID: \"4249b8fd-e594-496c-82f5-1c37fa2b6b1d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-csqq9" Nov 02 17:36:16.380124 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.379966 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nlhp\" (UniqueName: \"kubernetes.io/projected/4249b8fd-e594-496c-82f5-1c37fa2b6b1d-kube-api-access-5nlhp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-csqq9\" (UID: \"4249b8fd-e594-496c-82f5-1c37fa2b6b1d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-csqq9" Nov 02 17:36:16.380314 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.380293 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4249b8fd-e594-496c-82f5-1c37fa2b6b1d-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-csqq9\" (UID: \"4249b8fd-e594-496c-82f5-1c37fa2b6b1d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-csqq9" Nov 02 17:36:16.388573 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.388511 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nlhp\" (UniqueName: \"kubernetes.io/projected/4249b8fd-e594-496c-82f5-1c37fa2b6b1d-kube-api-access-5nlhp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-csqq9\" (UID: \"4249b8fd-e594-496c-82f5-1c37fa2b6b1d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-csqq9" Nov 02 17:36:16.516694 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.516662 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-csqq9" Nov 02 17:36:16.809106 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:16.809073 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-csqq9"] Nov 02 17:36:16.813451 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:36:16.813424 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4249b8fd_e594_496c_82f5_1c37fa2b6b1d.slice/crio-6e1082fa38023deabd67cb4c99a5aff1e263c87791f54d9e4147facfe354d5ca WatchSource:0}: Error finding container 6e1082fa38023deabd67cb4c99a5aff1e263c87791f54d9e4147facfe354d5ca: Status 404 returned error can't find the container with id 6e1082fa38023deabd67cb4c99a5aff1e263c87791f54d9e4147facfe354d5ca Nov 02 17:36:17.211741 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:17.211703 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-csqq9" event={"ID":"4249b8fd-e594-496c-82f5-1c37fa2b6b1d","Type":"ContainerStarted","Data":"6e1082fa38023deabd67cb4c99a5aff1e263c87791f54d9e4147facfe354d5ca"} Nov 02 17:36:19.135104 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.135073 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w"] Nov 02 17:36:19.207212 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.207179 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w"] Nov 02 17:36:19.207393 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.207316 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.281222 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.281186 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 17:36:19.281534 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.281514 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 17:36:19.282201 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.282166 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 17:36:19.282201 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.282192 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 17:36:19.282384 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.282215 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 17:36:19.282384 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.282305 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-z8prg\"" Nov 02 17:36:19.306365 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.306331 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/40b11c81-5baa-481c-b138-3ddc1f2e633c-metrics-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-xc65w\" (UID: \"40b11c81-5baa-481c-b138-3ddc1f2e633c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.306524 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.306379 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/40b11c81-5baa-481c-b138-3ddc1f2e633c-manifests\") pod \"opendatahub-operator-controller-manager-789486f7c8-xc65w\" (UID: \"40b11c81-5baa-481c-b138-3ddc1f2e633c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.306597 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.306514 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40b11c81-5baa-481c-b138-3ddc1f2e633c-webhook-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-xc65w\" (UID: \"40b11c81-5baa-481c-b138-3ddc1f2e633c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.306597 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.306566 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/40b11c81-5baa-481c-b138-3ddc1f2e633c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-xc65w\" (UID: \"40b11c81-5baa-481c-b138-3ddc1f2e633c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.306739 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.306603 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40b11c81-5baa-481c-b138-3ddc1f2e633c-tmp\") pod \"opendatahub-operator-controller-manager-789486f7c8-xc65w\" (UID: \"40b11c81-5baa-481c-b138-3ddc1f2e633c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.306739 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.306694 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzt28\" (UniqueName: \"kubernetes.io/projected/40b11c81-5baa-481c-b138-3ddc1f2e633c-kube-api-access-rzt28\") pod \"opendatahub-operator-controller-manager-789486f7c8-xc65w\" (UID: \"40b11c81-5baa-481c-b138-3ddc1f2e633c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.408010 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.407943 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/40b11c81-5baa-481c-b138-3ddc1f2e633c-metrics-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-xc65w\" (UID: \"40b11c81-5baa-481c-b138-3ddc1f2e633c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.408010 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.407974 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/40b11c81-5baa-481c-b138-3ddc1f2e633c-manifests\") pod \"opendatahub-operator-controller-manager-789486f7c8-xc65w\" (UID: \"40b11c81-5baa-481c-b138-3ddc1f2e633c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.408010 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.408000 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40b11c81-5baa-481c-b138-3ddc1f2e633c-webhook-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-xc65w\" (UID: \"40b11c81-5baa-481c-b138-3ddc1f2e633c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.408275 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.408029 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/40b11c81-5baa-481c-b138-3ddc1f2e633c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-xc65w\" (UID: \"40b11c81-5baa-481c-b138-3ddc1f2e633c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.408275 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.408068 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40b11c81-5baa-481c-b138-3ddc1f2e633c-tmp\") pod \"opendatahub-operator-controller-manager-789486f7c8-xc65w\" (UID: \"40b11c81-5baa-481c-b138-3ddc1f2e633c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.408275 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.408123 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzt28\" (UniqueName: \"kubernetes.io/projected/40b11c81-5baa-481c-b138-3ddc1f2e633c-kube-api-access-rzt28\") pod \"opendatahub-operator-controller-manager-789486f7c8-xc65w\" (UID: \"40b11c81-5baa-481c-b138-3ddc1f2e633c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.408423 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.408400 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/40b11c81-5baa-481c-b138-3ddc1f2e633c-manifests\") pod \"opendatahub-operator-controller-manager-789486f7c8-xc65w\" (UID: \"40b11c81-5baa-481c-b138-3ddc1f2e633c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.410567 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.410542 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40b11c81-5baa-481c-b138-3ddc1f2e633c-tmp\") pod \"opendatahub-operator-controller-manager-789486f7c8-xc65w\" (UID: \"40b11c81-5baa-481c-b138-3ddc1f2e633c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.410742 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.410723 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/40b11c81-5baa-481c-b138-3ddc1f2e633c-metrics-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-xc65w\" (UID: \"40b11c81-5baa-481c-b138-3ddc1f2e633c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.410809 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.410723 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/40b11c81-5baa-481c-b138-3ddc1f2e633c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-xc65w\" (UID: \"40b11c81-5baa-481c-b138-3ddc1f2e633c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.410809 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.410769 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40b11c81-5baa-481c-b138-3ddc1f2e633c-webhook-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-xc65w\" (UID: \"40b11c81-5baa-481c-b138-3ddc1f2e633c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.417390 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.417365 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzt28\" (UniqueName: \"kubernetes.io/projected/40b11c81-5baa-481c-b138-3ddc1f2e633c-kube-api-access-rzt28\") pod \"opendatahub-operator-controller-manager-789486f7c8-xc65w\" (UID: \"40b11c81-5baa-481c-b138-3ddc1f2e633c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.588001 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.587963 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:19.957460 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:36:19.957419 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b11c81_5baa_481c_b138_3ddc1f2e633c.slice/crio-345f2ccdeb78ab22ada4af4a2365174890328b27e7874c002b476fdf32f3eb1d WatchSource:0}: Error finding container 345f2ccdeb78ab22ada4af4a2365174890328b27e7874c002b476fdf32f3eb1d: Status 404 returned error can't find the container with id 345f2ccdeb78ab22ada4af4a2365174890328b27e7874c002b476fdf32f3eb1d Nov 02 17:36:19.957778 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:19.957746 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w"] Nov 02 17:36:20.231833 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:20.231742 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-csqq9" event={"ID":"4249b8fd-e594-496c-82f5-1c37fa2b6b1d","Type":"ContainerStarted","Data":"82c2fcc53bbd80f7e598667a29fd3d07865ab17c0463f39dfe673278188bc90d"} Nov 02 17:36:20.233065 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:20.233039 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" event={"ID":"40b11c81-5baa-481c-b138-3ddc1f2e633c","Type":"ContainerStarted","Data":"345f2ccdeb78ab22ada4af4a2365174890328b27e7874c002b476fdf32f3eb1d"} Nov 02 17:36:20.256886 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:20.256688 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-csqq9" podStartSLOduration=1.126506958 podStartE2EDuration="4.256672538s" totalImagesPullingTime="3.13016558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:20.256601074 +0000 UTC m=+144.947655087" watchObservedRunningTime="2026-11-02 17:36:20.256672538 +0000 UTC m=+144.947726551" Nov 02 17:36:21.268003 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:21.267965 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-h95t4"] Nov 02 17:36:21.271567 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:21.271462 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h95t4" Nov 02 17:36:21.282186 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:21.282013 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 17:36:21.282186 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:21.282155 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 17:36:21.285460 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:21.285424 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-vbrbt\"" Nov 02 17:36:21.289385 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:21.289352 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-h95t4"] Nov 02 17:36:21.428528 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:21.428491 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0622e414-7cf5-47f3-b8d1-1b79d6e73750-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-h95t4\" (UID: \"0622e414-7cf5-47f3-b8d1-1b79d6e73750\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h95t4" Nov 02 17:36:21.428718 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:21.428581 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7kw6\" (UniqueName: \"kubernetes.io/projected/0622e414-7cf5-47f3-b8d1-1b79d6e73750-kube-api-access-d7kw6\") pod \"cert-manager-webhook-5dbbbf75cc-h95t4\" (UID: \"0622e414-7cf5-47f3-b8d1-1b79d6e73750\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h95t4" Nov 02 17:36:21.530066 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:21.529991 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0622e414-7cf5-47f3-b8d1-1b79d6e73750-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-h95t4\" (UID: \"0622e414-7cf5-47f3-b8d1-1b79d6e73750\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h95t4" Nov 02 17:36:21.530066 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:21.530049 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7kw6\" (UniqueName: \"kubernetes.io/projected/0622e414-7cf5-47f3-b8d1-1b79d6e73750-kube-api-access-d7kw6\") pod \"cert-manager-webhook-5dbbbf75cc-h95t4\" (UID: \"0622e414-7cf5-47f3-b8d1-1b79d6e73750\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h95t4" Nov 02 17:36:21.544279 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:21.544248 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0622e414-7cf5-47f3-b8d1-1b79d6e73750-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-h95t4\" (UID: \"0622e414-7cf5-47f3-b8d1-1b79d6e73750\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h95t4" Nov 02 17:36:21.544892 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:21.544874 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7kw6\" (UniqueName: \"kubernetes.io/projected/0622e414-7cf5-47f3-b8d1-1b79d6e73750-kube-api-access-d7kw6\") pod \"cert-manager-webhook-5dbbbf75cc-h95t4\" (UID: \"0622e414-7cf5-47f3-b8d1-1b79d6e73750\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h95t4" Nov 02 17:36:21.592435 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:21.592405 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h95t4" Nov 02 17:36:21.889768 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:21.889737 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-h95t4"] Nov 02 17:36:21.892515 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:36:21.892458 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0622e414_7cf5_47f3_b8d1_1b79d6e73750.slice/crio-127dbeb2a6776ed33c694e04f4a2f75fe1830a2ceaca5f5bf7fb25aea5f20ad0 WatchSource:0}: Error finding container 127dbeb2a6776ed33c694e04f4a2f75fe1830a2ceaca5f5bf7fb25aea5f20ad0: Status 404 returned error can't find the container with id 127dbeb2a6776ed33c694e04f4a2f75fe1830a2ceaca5f5bf7fb25aea5f20ad0 Nov 02 17:36:22.008774 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:22.008740 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-vh8nk"] Nov 02 17:36:22.011582 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:22.011563 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vh8nk" Nov 02 17:36:22.020423 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:22.020399 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:36:22.020533 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:22.020435 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 17:36:22.021914 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:22.021895 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-9nnl6\"" Nov 02 17:36:22.025065 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:22.025044 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-vh8nk"] Nov 02 17:36:22.136537 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:22.136502 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/063375fe-6941-4323-b867-5d82826b36da-tmp\") pod \"jobset-operator-674cd59dc5-vh8nk\" (UID: \"063375fe-6941-4323-b867-5d82826b36da\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vh8nk" Nov 02 17:36:22.136537 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:22.136549 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-564nh\" (UniqueName: \"kubernetes.io/projected/063375fe-6941-4323-b867-5d82826b36da-kube-api-access-564nh\") pod \"jobset-operator-674cd59dc5-vh8nk\" (UID: \"063375fe-6941-4323-b867-5d82826b36da\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vh8nk" Nov 02 17:36:22.237902 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:22.237865 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/063375fe-6941-4323-b867-5d82826b36da-tmp\") pod \"jobset-operator-674cd59dc5-vh8nk\" (UID: \"063375fe-6941-4323-b867-5d82826b36da\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vh8nk" Nov 02 17:36:22.238095 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:22.237919 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-564nh\" (UniqueName: \"kubernetes.io/projected/063375fe-6941-4323-b867-5d82826b36da-kube-api-access-564nh\") pod \"jobset-operator-674cd59dc5-vh8nk\" (UID: \"063375fe-6941-4323-b867-5d82826b36da\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vh8nk" Nov 02 17:36:22.238393 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:22.238361 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/063375fe-6941-4323-b867-5d82826b36da-tmp\") pod \"jobset-operator-674cd59dc5-vh8nk\" (UID: \"063375fe-6941-4323-b867-5d82826b36da\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vh8nk" Nov 02 17:36:22.245381 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:22.245352 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h95t4" event={"ID":"0622e414-7cf5-47f3-b8d1-1b79d6e73750","Type":"ContainerStarted","Data":"127dbeb2a6776ed33c694e04f4a2f75fe1830a2ceaca5f5bf7fb25aea5f20ad0"} Nov 02 17:36:22.256919 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:22.256897 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-564nh\" (UniqueName: \"kubernetes.io/projected/063375fe-6941-4323-b867-5d82826b36da-kube-api-access-564nh\") pod \"jobset-operator-674cd59dc5-vh8nk\" (UID: \"063375fe-6941-4323-b867-5d82826b36da\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vh8nk" Nov 02 17:36:22.325232 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:22.325197 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vh8nk" Nov 02 17:36:22.634392 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:22.634365 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-vh8nk"] Nov 02 17:36:22.636447 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:36:22.636416 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod063375fe_6941_4323_b867_5d82826b36da.slice/crio-863552e71e981f21a04a96d1af29269359523c45cb251709a863ba9c3e4c8c71 WatchSource:0}: Error finding container 863552e71e981f21a04a96d1af29269359523c45cb251709a863ba9c3e4c8c71: Status 404 returned error can't find the container with id 863552e71e981f21a04a96d1af29269359523c45cb251709a863ba9c3e4c8c71 Nov 02 17:36:23.259006 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:23.258966 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vh8nk" event={"ID":"063375fe-6941-4323-b867-5d82826b36da","Type":"ContainerStarted","Data":"863552e71e981f21a04a96d1af29269359523c45cb251709a863ba9c3e4c8c71"} Nov 02 17:36:23.260799 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:23.260770 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" event={"ID":"40b11c81-5baa-481c-b138-3ddc1f2e633c","Type":"ContainerStarted","Data":"1745baa1162ba67d1e084a926b7f0aac00daf6f1fc40af12e5002c0f446cdadd"} Nov 02 17:36:24.267968 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:24.267920 5601 generic.go:356] "Generic (PLEG): container finished" podID="40b11c81-5baa-481c-b138-3ddc1f2e633c" containerID="1745baa1162ba67d1e084a926b7f0aac00daf6f1fc40af12e5002c0f446cdadd" exitCode=0 Nov 02 17:36:24.268463 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:24.267975 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" event={"ID":"40b11c81-5baa-481c-b138-3ddc1f2e633c","Type":"ContainerDied","Data":"1745baa1162ba67d1e084a926b7f0aac00daf6f1fc40af12e5002c0f446cdadd"} Nov 02 17:36:25.279314 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:25.279209 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h95t4" event={"ID":"0622e414-7cf5-47f3-b8d1-1b79d6e73750","Type":"ContainerStarted","Data":"119f9cd9798596e9755dc9f0dcfc0aef6dcb1b209c3d752f0d711f78a20ade12"} Nov 02 17:36:25.279314 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:25.279264 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h95t4" Nov 02 17:36:25.281149 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:25.281121 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vh8nk" event={"ID":"063375fe-6941-4323-b867-5d82826b36da","Type":"ContainerStarted","Data":"095b142de21eebb0c02148dfd9121a897698e5d72ba91ad2ea3027d454c8403b"} Nov 02 17:36:25.283158 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:25.283136 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" event={"ID":"40b11c81-5baa-481c-b138-3ddc1f2e633c","Type":"ContainerStarted","Data":"0ec580d1b0d75984149ce59f1077fbbb6b7249bc5f76e99d90a73f7a36725b29"} Nov 02 17:36:25.283271 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:25.283232 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:25.299347 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:25.299303 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h95t4" podStartSLOduration=1.223398403 podStartE2EDuration="4.299291159s" totalImagesPullingTime="3.075892756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:25.298052865 +0000 UTC m=+149.989106876" watchObservedRunningTime="2026-11-02 17:36:25.299291159 +0000 UTC m=+149.990345171" Nov 02 17:36:25.345529 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:25.345479 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" podStartSLOduration=2.241797837 podStartE2EDuration="6.345460948s" totalImagesPullingTime="3.421092904s" totalInitContainerRuntime="682.570207ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:25.343419953 +0000 UTC m=+150.034473965" watchObservedRunningTime="2026-11-02 17:36:25.345460948 +0000 UTC m=+150.036514964" Nov 02 17:36:25.363828 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:25.363786 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vh8nk" podStartSLOduration=2.06590572 podStartE2EDuration="4.363769887s" totalImagesPullingTime="2.297864167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:25.363489253 +0000 UTC m=+150.054543267" watchObservedRunningTime="2026-11-02 17:36:25.363769887 +0000 UTC m=+150.054823900" Nov 02 17:36:29.337765 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:29.337725 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-zqnb2"] Nov 02 17:36:29.341957 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:29.341935 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zqnb2" Nov 02 17:36:29.352650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:29.352611 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5qs6c\"" Nov 02 17:36:29.364711 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:29.364680 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-zqnb2"] Nov 02 17:36:29.504652 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:29.504600 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d3f424f-6bfa-443e-a3e7-a250558df591-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-zqnb2\" (UID: \"4d3f424f-6bfa-443e-a3e7-a250558df591\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zqnb2" Nov 02 17:36:29.504804 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:29.504660 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pktgj\" (UniqueName: \"kubernetes.io/projected/4d3f424f-6bfa-443e-a3e7-a250558df591-kube-api-access-pktgj\") pod \"cert-manager-cainjector-5c57bbd4fc-zqnb2\" (UID: \"4d3f424f-6bfa-443e-a3e7-a250558df591\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zqnb2" Nov 02 17:36:29.605496 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:29.605411 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d3f424f-6bfa-443e-a3e7-a250558df591-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-zqnb2\" (UID: \"4d3f424f-6bfa-443e-a3e7-a250558df591\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zqnb2" Nov 02 17:36:29.605496 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:29.605467 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pktgj\" (UniqueName: \"kubernetes.io/projected/4d3f424f-6bfa-443e-a3e7-a250558df591-kube-api-access-pktgj\") pod \"cert-manager-cainjector-5c57bbd4fc-zqnb2\" (UID: \"4d3f424f-6bfa-443e-a3e7-a250558df591\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zqnb2" Nov 02 17:36:29.614989 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:29.614961 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d3f424f-6bfa-443e-a3e7-a250558df591-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-zqnb2\" (UID: \"4d3f424f-6bfa-443e-a3e7-a250558df591\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zqnb2" Nov 02 17:36:29.615079 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:29.615017 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pktgj\" (UniqueName: \"kubernetes.io/projected/4d3f424f-6bfa-443e-a3e7-a250558df591-kube-api-access-pktgj\") pod \"cert-manager-cainjector-5c57bbd4fc-zqnb2\" (UID: \"4d3f424f-6bfa-443e-a3e7-a250558df591\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zqnb2" Nov 02 17:36:29.651991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:29.651969 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zqnb2" Nov 02 17:36:29.927830 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:29.927804 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-zqnb2"] Nov 02 17:36:29.930555 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:36:29.930527 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d3f424f_6bfa_443e_a3e7_a250558df591.slice/crio-8b95d786c52b2c5ba948fab824f277f455e90c362e0e67b05f29c39d655241db WatchSource:0}: Error finding container 8b95d786c52b2c5ba948fab824f277f455e90c362e0e67b05f29c39d655241db: Status 404 returned error can't find the container with id 8b95d786c52b2c5ba948fab824f277f455e90c362e0e67b05f29c39d655241db Nov 02 17:36:30.305955 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:30.305869 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zqnb2" event={"ID":"4d3f424f-6bfa-443e-a3e7-a250558df591","Type":"ContainerStarted","Data":"1d2a80e38c3353c9f72f05aee5cd5f17aa2cfd180856fce21d5fbabae6bf69d3"} Nov 02 17:36:30.305955 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:30.305911 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zqnb2" event={"ID":"4d3f424f-6bfa-443e-a3e7-a250558df591","Type":"ContainerStarted","Data":"8b95d786c52b2c5ba948fab824f277f455e90c362e0e67b05f29c39d655241db"} Nov 02 17:36:30.323751 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:30.323695 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zqnb2" podStartSLOduration=1.323680244 podStartE2EDuration="1.323680244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:30.3226973 +0000 UTC m=+155.013751313" watchObservedRunningTime="2026-11-02 17:36:30.323680244 +0000 UTC m=+155.014734246" Nov 02 17:36:31.290245 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:31.290209 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h95t4" Nov 02 17:36:32.451493 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:32.451459 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-ztqjp"] Nov 02 17:36:32.453869 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:32.453850 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-ztqjp" Nov 02 17:36:32.458860 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:32.458839 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-v69qp\"" Nov 02 17:36:32.463722 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:32.463695 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-ztqjp"] Nov 02 17:36:32.530848 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:32.530809 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np8ts\" (UniqueName: \"kubernetes.io/projected/a53277ea-04b8-4915-b5d9-c4b320eeae0f-kube-api-access-np8ts\") pod \"cert-manager-597b95cb84-ztqjp\" (UID: \"a53277ea-04b8-4915-b5d9-c4b320eeae0f\") " pod="cert-manager/cert-manager-597b95cb84-ztqjp" Nov 02 17:36:32.530848 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:32.530850 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a53277ea-04b8-4915-b5d9-c4b320eeae0f-bound-sa-token\") pod \"cert-manager-597b95cb84-ztqjp\" (UID: \"a53277ea-04b8-4915-b5d9-c4b320eeae0f\") " pod="cert-manager/cert-manager-597b95cb84-ztqjp" Nov 02 17:36:32.631750 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:32.631709 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np8ts\" (UniqueName: \"kubernetes.io/projected/a53277ea-04b8-4915-b5d9-c4b320eeae0f-kube-api-access-np8ts\") pod \"cert-manager-597b95cb84-ztqjp\" (UID: \"a53277ea-04b8-4915-b5d9-c4b320eeae0f\") " pod="cert-manager/cert-manager-597b95cb84-ztqjp" Nov 02 17:36:32.631750 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:32.631756 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a53277ea-04b8-4915-b5d9-c4b320eeae0f-bound-sa-token\") pod \"cert-manager-597b95cb84-ztqjp\" (UID: \"a53277ea-04b8-4915-b5d9-c4b320eeae0f\") " pod="cert-manager/cert-manager-597b95cb84-ztqjp" Nov 02 17:36:32.643188 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:32.643159 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a53277ea-04b8-4915-b5d9-c4b320eeae0f-bound-sa-token\") pod \"cert-manager-597b95cb84-ztqjp\" (UID: \"a53277ea-04b8-4915-b5d9-c4b320eeae0f\") " pod="cert-manager/cert-manager-597b95cb84-ztqjp" Nov 02 17:36:32.643434 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:32.643410 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np8ts\" (UniqueName: \"kubernetes.io/projected/a53277ea-04b8-4915-b5d9-c4b320eeae0f-kube-api-access-np8ts\") pod \"cert-manager-597b95cb84-ztqjp\" (UID: \"a53277ea-04b8-4915-b5d9-c4b320eeae0f\") " pod="cert-manager/cert-manager-597b95cb84-ztqjp" Nov 02 17:36:32.766230 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:32.766137 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-ztqjp" Nov 02 17:36:33.040162 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:33.040137 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-ztqjp"] Nov 02 17:36:33.042457 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:36:33.042430 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda53277ea_04b8_4915_b5d9_c4b320eeae0f.slice/crio-cb6eb6597ba417b2d352ecd0e594eb167d5293fc75fc3a74e3497a26cd0a3f67 WatchSource:0}: Error finding container cb6eb6597ba417b2d352ecd0e594eb167d5293fc75fc3a74e3497a26cd0a3f67: Status 404 returned error can't find the container with id cb6eb6597ba417b2d352ecd0e594eb167d5293fc75fc3a74e3497a26cd0a3f67 Nov 02 17:36:33.320861 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:33.320775 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-ztqjp" event={"ID":"a53277ea-04b8-4915-b5d9-c4b320eeae0f","Type":"ContainerStarted","Data":"9adef95147c83b2a69331a13f1800d6855f6d895d0abddcf860e6dddf5b497d5"} Nov 02 17:36:33.320861 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:33.320816 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-ztqjp" event={"ID":"a53277ea-04b8-4915-b5d9-c4b320eeae0f","Type":"ContainerStarted","Data":"cb6eb6597ba417b2d352ecd0e594eb167d5293fc75fc3a74e3497a26cd0a3f67"} Nov 02 17:36:33.339857 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:33.339804 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-ztqjp" podStartSLOduration=1.339787675 podStartE2EDuration="1.339787675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:33.339253592 +0000 UTC m=+158.030307604" watchObservedRunningTime="2026-11-02 17:36:33.339787675 +0000 UTC m=+158.030841692" Nov 02 17:36:34.483586 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:36:34.483548 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:36:36.290195 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:36.290160 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-xc65w" Nov 02 17:36:53.125187 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:53.125148 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cf9f7b655-m7r89"] Nov 02 17:36:53.132529 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:53.132498 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-m7r89" Nov 02 17:36:53.147844 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:53.147815 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cf9f7b655-m7r89"] Nov 02 17:36:53.204431 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:53.204404 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d1ee04f-2201-4da0-88ff-f06683fc6006-tmp\") pod \"kube-auth-proxy-5cf9f7b655-m7r89\" (UID: \"9d1ee04f-2201-4da0-88ff-f06683fc6006\") " pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-m7r89" Nov 02 17:36:53.204643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:53.204447 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnzvv\" (UniqueName: \"kubernetes.io/projected/9d1ee04f-2201-4da0-88ff-f06683fc6006-kube-api-access-fnzvv\") pod \"kube-auth-proxy-5cf9f7b655-m7r89\" (UID: \"9d1ee04f-2201-4da0-88ff-f06683fc6006\") " pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-m7r89" Nov 02 17:36:53.204643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:53.204480 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d1ee04f-2201-4da0-88ff-f06683fc6006-tls-certs\") pod \"kube-auth-proxy-5cf9f7b655-m7r89\" (UID: \"9d1ee04f-2201-4da0-88ff-f06683fc6006\") " pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-m7r89" Nov 02 17:36:53.227784 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:53.227756 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 17:36:53.227934 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:53.227884 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7krc4\"" Nov 02 17:36:53.228008 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:53.227962 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 17:36:53.305945 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:53.305912 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d1ee04f-2201-4da0-88ff-f06683fc6006-tmp\") pod \"kube-auth-proxy-5cf9f7b655-m7r89\" (UID: \"9d1ee04f-2201-4da0-88ff-f06683fc6006\") " pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-m7r89" Nov 02 17:36:53.306087 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:53.305954 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnzvv\" (UniqueName: \"kubernetes.io/projected/9d1ee04f-2201-4da0-88ff-f06683fc6006-kube-api-access-fnzvv\") pod \"kube-auth-proxy-5cf9f7b655-m7r89\" (UID: \"9d1ee04f-2201-4da0-88ff-f06683fc6006\") " pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-m7r89" Nov 02 17:36:53.306087 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:53.305997 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d1ee04f-2201-4da0-88ff-f06683fc6006-tls-certs\") pod \"kube-auth-proxy-5cf9f7b655-m7r89\" (UID: \"9d1ee04f-2201-4da0-88ff-f06683fc6006\") " pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-m7r89" Nov 02 17:36:53.308368 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:53.308346 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d1ee04f-2201-4da0-88ff-f06683fc6006-tmp\") pod \"kube-auth-proxy-5cf9f7b655-m7r89\" (UID: \"9d1ee04f-2201-4da0-88ff-f06683fc6006\") " pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-m7r89" Nov 02 17:36:53.308507 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:53.308488 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d1ee04f-2201-4da0-88ff-f06683fc6006-tls-certs\") pod \"kube-auth-proxy-5cf9f7b655-m7r89\" (UID: \"9d1ee04f-2201-4da0-88ff-f06683fc6006\") " pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-m7r89" Nov 02 17:36:53.333875 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:53.333842 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnzvv\" (UniqueName: \"kubernetes.io/projected/9d1ee04f-2201-4da0-88ff-f06683fc6006-kube-api-access-fnzvv\") pod \"kube-auth-proxy-5cf9f7b655-m7r89\" (UID: \"9d1ee04f-2201-4da0-88ff-f06683fc6006\") " pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-m7r89" Nov 02 17:36:53.444400 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:53.444370 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-m7r89" Nov 02 17:36:53.734066 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:53.733976 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cf9f7b655-m7r89"] Nov 02 17:36:53.737239 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:36:53.737203 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1ee04f_2201_4da0_88ff_f06683fc6006.slice/crio-fcb7b1a42ad959d8953fbaa6431ec0f90aee48d33707a32ea07fed0be7fe454f WatchSource:0}: Error finding container fcb7b1a42ad959d8953fbaa6431ec0f90aee48d33707a32ea07fed0be7fe454f: Status 404 returned error can't find the container with id fcb7b1a42ad959d8953fbaa6431ec0f90aee48d33707a32ea07fed0be7fe454f Nov 02 17:36:54.102296 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:54.102217 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-thb2f"] Nov 02 17:36:54.106699 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:54.106680 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-thb2f" Nov 02 17:36:54.113926 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:54.113907 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hhm7t\"" Nov 02 17:36:54.119095 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:54.119073 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-thb2f"] Nov 02 17:36:54.169640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:54.169530 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 17:36:54.170013 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:54.169574 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 17:36:54.170013 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:54.169717 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 17:36:54.202978 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:54.202948 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-thb2f"] Nov 02 17:36:54.213603 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:54.213572 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94r29\" (UniqueName: \"kubernetes.io/projected/a80e4d21-6960-4357-ad8f-77c85f7ec6e9-kube-api-access-94r29\") pod \"limitador-limitador-74dfcfb894-thb2f\" (UID: \"a80e4d21-6960-4357-ad8f-77c85f7ec6e9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-thb2f" Nov 02 17:36:54.213763 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:54.213656 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a80e4d21-6960-4357-ad8f-77c85f7ec6e9-config-file\") pod \"limitador-limitador-74dfcfb894-thb2f\" (UID: \"a80e4d21-6960-4357-ad8f-77c85f7ec6e9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-thb2f" Nov 02 17:36:54.314344 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:54.314313 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a80e4d21-6960-4357-ad8f-77c85f7ec6e9-config-file\") pod \"limitador-limitador-74dfcfb894-thb2f\" (UID: \"a80e4d21-6960-4357-ad8f-77c85f7ec6e9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-thb2f" Nov 02 17:36:54.314509 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:54.314401 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94r29\" (UniqueName: \"kubernetes.io/projected/a80e4d21-6960-4357-ad8f-77c85f7ec6e9-kube-api-access-94r29\") pod \"limitador-limitador-74dfcfb894-thb2f\" (UID: \"a80e4d21-6960-4357-ad8f-77c85f7ec6e9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-thb2f" Nov 02 17:36:54.314958 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:54.314933 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a80e4d21-6960-4357-ad8f-77c85f7ec6e9-config-file\") pod \"limitador-limitador-74dfcfb894-thb2f\" (UID: \"a80e4d21-6960-4357-ad8f-77c85f7ec6e9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-thb2f" Nov 02 17:36:54.334382 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:54.334354 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94r29\" (UniqueName: \"kubernetes.io/projected/a80e4d21-6960-4357-ad8f-77c85f7ec6e9-kube-api-access-94r29\") pod \"limitador-limitador-74dfcfb894-thb2f\" (UID: \"a80e4d21-6960-4357-ad8f-77c85f7ec6e9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-thb2f" Nov 02 17:36:54.417442 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:54.417371 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-m7r89" event={"ID":"9d1ee04f-2201-4da0-88ff-f06683fc6006","Type":"ContainerStarted","Data":"fcb7b1a42ad959d8953fbaa6431ec0f90aee48d33707a32ea07fed0be7fe454f"} Nov 02 17:36:54.419686 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:54.419656 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-thb2f" Nov 02 17:36:54.717147 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:54.716875 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-thb2f"] Nov 02 17:36:54.720802 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:36:54.720766 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80e4d21_6960_4357_ad8f_77c85f7ec6e9.slice/crio-0e2ef1cea924d44624560a806357bc9a86acf108f4e46c8bb1d474280db0ee18 WatchSource:0}: Error finding container 0e2ef1cea924d44624560a806357bc9a86acf108f4e46c8bb1d474280db0ee18: Status 404 returned error can't find the container with id 0e2ef1cea924d44624560a806357bc9a86acf108f4e46c8bb1d474280db0ee18 Nov 02 17:36:55.430436 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:55.430396 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-thb2f" event={"ID":"a80e4d21-6960-4357-ad8f-77c85f7ec6e9","Type":"ContainerStarted","Data":"0e2ef1cea924d44624560a806357bc9a86acf108f4e46c8bb1d474280db0ee18"} Nov 02 17:36:55.453985 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:55.453952 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-d7z2j"] Nov 02 17:36:55.464419 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:55.464392 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-d7z2j" Nov 02 17:36:55.470249 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:55.470197 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-d7z2j"] Nov 02 17:36:55.476875 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:55.476839 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qjnd2\"" Nov 02 17:36:55.528627 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:55.528574 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psgqw\" (UniqueName: \"kubernetes.io/projected/f5865ba9-7951-4082-b7b0-743110225f59-kube-api-access-psgqw\") pod \"authorino-56c6c9ddd5-d7z2j\" (UID: \"f5865ba9-7951-4082-b7b0-743110225f59\") " pod="kuadrant-system/authorino-56c6c9ddd5-d7z2j" Nov 02 17:36:55.629791 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:55.629760 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psgqw\" (UniqueName: \"kubernetes.io/projected/f5865ba9-7951-4082-b7b0-743110225f59-kube-api-access-psgqw\") pod \"authorino-56c6c9ddd5-d7z2j\" (UID: \"f5865ba9-7951-4082-b7b0-743110225f59\") " pod="kuadrant-system/authorino-56c6c9ddd5-d7z2j" Nov 02 17:36:55.639262 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:55.639231 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psgqw\" (UniqueName: \"kubernetes.io/projected/f5865ba9-7951-4082-b7b0-743110225f59-kube-api-access-psgqw\") pod \"authorino-56c6c9ddd5-d7z2j\" (UID: \"f5865ba9-7951-4082-b7b0-743110225f59\") " pod="kuadrant-system/authorino-56c6c9ddd5-d7z2j" Nov 02 17:36:55.780502 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:55.780425 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-d7z2j" Nov 02 17:36:56.096270 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:56.096236 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-d7z2j"] Nov 02 17:36:56.437656 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:56.437600 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-d7z2j" event={"ID":"f5865ba9-7951-4082-b7b0-743110225f59","Type":"ContainerStarted","Data":"a0d91b1cd66876cbc5360e8e4cd98a661e51bdc004e0a09d38deab7c08a525a0"} Nov 02 17:36:56.822772 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:56.822744 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 17:36:57.449042 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:57.449003 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-m7r89" event={"ID":"9d1ee04f-2201-4da0-88ff-f06683fc6006","Type":"ContainerStarted","Data":"6a985b1ffbb080f073a907b718154be4b15ac165967d17c35d20ba557a5a5db0"} Nov 02 17:36:57.487148 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:57.487092 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cf9f7b655-m7r89" podStartSLOduration=1.406767035 podStartE2EDuration="4.487077932s" totalImagesPullingTime="3.080310897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:57.482743609 +0000 UTC m=+182.173797621" watchObservedRunningTime="2026-11-02 17:36:57.487077932 +0000 UTC m=+182.178131942" Nov 02 17:36:59.459143 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:59.459111 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-thb2f" event={"ID":"a80e4d21-6960-4357-ad8f-77c85f7ec6e9","Type":"ContainerStarted","Data":"ab87b721d811a5fd733f8cf0911a02b336f0722d7b14b8159663739c0854ec57"} Nov 02 17:36:59.459575 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:59.459243 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-thb2f" Nov 02 17:36:59.460655 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:59.460633 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-d7z2j" event={"ID":"f5865ba9-7951-4082-b7b0-743110225f59","Type":"ContainerStarted","Data":"c4cd5fd106ee4759b4582d3cab2be9d1189d745d61fdeaaa7a49a4de129fbc40"} Nov 02 17:36:59.477372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:59.477330 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-thb2f" podStartSLOduration=1.702472455 podStartE2EDuration="5.477314687s" totalImagesPullingTime="3.774842232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:59.476469426 +0000 UTC m=+184.167523464" watchObservedRunningTime="2026-11-02 17:36:59.477314687 +0000 UTC m=+184.168368699" Nov 02 17:36:59.491268 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:36:59.491227 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-d7z2j" podStartSLOduration=1.710477703 podStartE2EDuration="4.491211801s" totalImagesPullingTime="2.780734098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:59.491099314 +0000 UTC m=+184.182153326" watchObservedRunningTime="2026-11-02 17:36:59.491211801 +0000 UTC m=+184.182265813" Nov 02 17:37:04.484025 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:37:04.483986 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:37:10.128874 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:10.128836 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-thb2f"] Nov 02 17:37:10.129450 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:10.129054 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-thb2f" podUID="a80e4d21-6960-4357-ad8f-77c85f7ec6e9" containerName="limitador" containerID="cri-o://ab87b721d811a5fd733f8cf0911a02b336f0722d7b14b8159663739c0854ec57" gracePeriod=30 Nov 02 17:37:10.130147 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:10.129738 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-thb2f" Nov 02 17:37:10.512837 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:10.512808 5601 generic.go:356] "Generic (PLEG): container finished" podID="a80e4d21-6960-4357-ad8f-77c85f7ec6e9" containerID="ab87b721d811a5fd733f8cf0911a02b336f0722d7b14b8159663739c0854ec57" exitCode=0 Nov 02 17:37:10.512947 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:10.512878 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-thb2f" event={"ID":"a80e4d21-6960-4357-ad8f-77c85f7ec6e9","Type":"ContainerDied","Data":"ab87b721d811a5fd733f8cf0911a02b336f0722d7b14b8159663739c0854ec57"} Nov 02 17:37:11.063797 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:11.063767 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-thb2f" Nov 02 17:37:11.178024 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:11.177985 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a80e4d21-6960-4357-ad8f-77c85f7ec6e9-kube-api-access-94r29\" (UniqueName: \"kubernetes.io/projected/a80e4d21-6960-4357-ad8f-77c85f7ec6e9-kube-api-access-94r29\") pod \"a80e4d21-6960-4357-ad8f-77c85f7ec6e9\" (UID: \"a80e4d21-6960-4357-ad8f-77c85f7ec6e9\") " Nov 02 17:37:11.178504 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:11.178041 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a80e4d21-6960-4357-ad8f-77c85f7ec6e9-config-file\" (UniqueName: \"kubernetes.io/configmap/a80e4d21-6960-4357-ad8f-77c85f7ec6e9-config-file\") pod \"a80e4d21-6960-4357-ad8f-77c85f7ec6e9\" (UID: \"a80e4d21-6960-4357-ad8f-77c85f7ec6e9\") " Nov 02 17:37:11.178504 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:11.178406 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80e4d21-6960-4357-ad8f-77c85f7ec6e9-config-file" pod "a80e4d21-6960-4357-ad8f-77c85f7ec6e9" (UID: "a80e4d21-6960-4357-ad8f-77c85f7ec6e9"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:37:11.180431 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:11.180406 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80e4d21-6960-4357-ad8f-77c85f7ec6e9-kube-api-access-94r29" pod "a80e4d21-6960-4357-ad8f-77c85f7ec6e9" (UID: "a80e4d21-6960-4357-ad8f-77c85f7ec6e9"). InnerVolumeSpecName "kube-api-access-94r29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:37:11.279299 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:11.279222 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94r29\" (UniqueName: \"kubernetes.io/projected/a80e4d21-6960-4357-ad8f-77c85f7ec6e9-kube-api-access-94r29\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:37:11.279299 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:11.279251 5601 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a80e4d21-6960-4357-ad8f-77c85f7ec6e9-config-file\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:37:11.518934 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:11.518901 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-thb2f" Nov 02 17:37:11.519106 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:11.518933 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-thb2f" event={"ID":"a80e4d21-6960-4357-ad8f-77c85f7ec6e9","Type":"ContainerDied","Data":"0e2ef1cea924d44624560a806357bc9a86acf108f4e46c8bb1d474280db0ee18"} Nov 02 17:37:11.519106 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:11.518995 5601 scope.go:122] "RemoveContainer" containerID="ab87b721d811a5fd733f8cf0911a02b336f0722d7b14b8159663739c0854ec57" Nov 02 17:37:11.537196 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:11.537131 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-thb2f"] Nov 02 17:37:11.541586 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:11.541562 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-thb2f"] Nov 02 17:37:13.424635 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:13.424577 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a80e4d21-6960-4357-ad8f-77c85f7ec6e9" path="/var/lib/kubelet/pods/a80e4d21-6960-4357-ad8f-77c85f7ec6e9/volumes" Nov 02 17:37:15.945552 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:15.945518 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-gk2ps"] Nov 02 17:37:15.945963 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:15.945860 5601 cpu_manager.go:414] "containerMap: removing container" podUID="a80e4d21-6960-4357-ad8f-77c85f7ec6e9" containerName="limitador" Nov 02 17:37:15.945963 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:15.945871 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a80e4d21-6960-4357-ad8f-77c85f7ec6e9" containerName="limitador" Nov 02 17:37:15.945963 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:15.945945 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="a80e4d21-6960-4357-ad8f-77c85f7ec6e9" containerName="limitador" Nov 02 17:37:15.948666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:15.948646 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gk2ps" Nov 02 17:37:15.963071 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:15.963043 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 17:37:15.963071 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:15.963047 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 17:37:15.963275 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:15.963112 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 17:37:15.963275 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:15.963139 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-cwwc8\"" Nov 02 17:37:15.963785 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:15.963765 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-gk2ps"] Nov 02 17:37:16.125914 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:16.125876 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79k5w\" (UniqueName: \"kubernetes.io/projected/0c86a124-ed3f-4592-a31a-09ad586482d1-kube-api-access-79k5w\") pod \"postgres-7cdbf6dc44-gk2ps\" (UID: \"0c86a124-ed3f-4592-a31a-09ad586482d1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gk2ps" Nov 02 17:37:16.126068 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:16.125954 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0c86a124-ed3f-4592-a31a-09ad586482d1-data\") pod \"postgres-7cdbf6dc44-gk2ps\" (UID: \"0c86a124-ed3f-4592-a31a-09ad586482d1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gk2ps" Nov 02 17:37:16.226539 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:16.226449 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79k5w\" (UniqueName: \"kubernetes.io/projected/0c86a124-ed3f-4592-a31a-09ad586482d1-kube-api-access-79k5w\") pod \"postgres-7cdbf6dc44-gk2ps\" (UID: \"0c86a124-ed3f-4592-a31a-09ad586482d1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gk2ps" Nov 02 17:37:16.226539 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:16.226530 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0c86a124-ed3f-4592-a31a-09ad586482d1-data\") pod \"postgres-7cdbf6dc44-gk2ps\" (UID: \"0c86a124-ed3f-4592-a31a-09ad586482d1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gk2ps" Nov 02 17:37:16.226958 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:16.226938 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0c86a124-ed3f-4592-a31a-09ad586482d1-data\") pod \"postgres-7cdbf6dc44-gk2ps\" (UID: \"0c86a124-ed3f-4592-a31a-09ad586482d1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gk2ps" Nov 02 17:37:16.237337 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:16.237294 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79k5w\" (UniqueName: \"kubernetes.io/projected/0c86a124-ed3f-4592-a31a-09ad586482d1-kube-api-access-79k5w\") pod \"postgres-7cdbf6dc44-gk2ps\" (UID: \"0c86a124-ed3f-4592-a31a-09ad586482d1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gk2ps" Nov 02 17:37:16.259468 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:16.259438 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gk2ps" Nov 02 17:37:16.542926 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:16.542902 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-gk2ps"] Nov 02 17:37:16.546129 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:37:16.546101 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c86a124_ed3f_4592_a31a_09ad586482d1.slice/crio-42c46c2c58a56af2dcfd80a5bae87ab3d57dad33614397a92c3309eb849fda0b WatchSource:0}: Error finding container 42c46c2c58a56af2dcfd80a5bae87ab3d57dad33614397a92c3309eb849fda0b: Status 404 returned error can't find the container with id 42c46c2c58a56af2dcfd80a5bae87ab3d57dad33614397a92c3309eb849fda0b Nov 02 17:37:17.551711 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:17.551675 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gk2ps" event={"ID":"0c86a124-ed3f-4592-a31a-09ad586482d1","Type":"ContainerStarted","Data":"42c46c2c58a56af2dcfd80a5bae87ab3d57dad33614397a92c3309eb849fda0b"} Nov 02 17:37:23.587044 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:23.587007 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gk2ps" event={"ID":"0c86a124-ed3f-4592-a31a-09ad586482d1","Type":"ContainerStarted","Data":"2c606536b3753a6d3e668994263f206d4934405d5d075e285ec616d98e56fd85"} Nov 02 17:37:23.587419 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:23.587134 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gk2ps" Nov 02 17:37:23.607142 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:23.607094 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gk2ps" podStartSLOduration=2.441093534 podStartE2EDuration="8.607082157s" totalImagesPullingTime="6.165988623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:23.606152392 +0000 UTC m=+208.297206403" watchObservedRunningTime="2026-11-02 17:37:23.607082157 +0000 UTC m=+208.298136167" Nov 02 17:37:29.619825 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:29.619791 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gk2ps" Nov 02 17:37:34.482401 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:37:34.482362 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:37:36.472076 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.472033 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78cc949cd9-vpjbq"] Nov 02 17:37:36.495418 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.495387 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78cc949cd9-vpjbq"] Nov 02 17:37:36.495571 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.495483 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78cc949cd9-vpjbq" Nov 02 17:37:36.500447 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.500419 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 17:37:36.501375 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.501165 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 17:37:36.501375 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.501281 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 17:37:36.501565 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.501396 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 17:37:36.503153 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.503129 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5hbfs\"" Nov 02 17:37:36.503271 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.503184 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 17:37:36.601239 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.601201 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/771b760a-10af-45ad-9c18-673e4ae6e6f9-webhook-cert\") pod \"maas-controller-78cc949cd9-vpjbq\" (UID: \"771b760a-10af-45ad-9c18-673e4ae6e6f9\") " pod="opendatahub/maas-controller-78cc949cd9-vpjbq" Nov 02 17:37:36.601239 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.601243 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/771b760a-10af-45ad-9c18-673e4ae6e6f9-metrics-certs\") pod \"maas-controller-78cc949cd9-vpjbq\" (UID: \"771b760a-10af-45ad-9c18-673e4ae6e6f9\") " pod="opendatahub/maas-controller-78cc949cd9-vpjbq" Nov 02 17:37:36.601447 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.601271 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7flm\" (UniqueName: \"kubernetes.io/projected/771b760a-10af-45ad-9c18-673e4ae6e6f9-kube-api-access-v7flm\") pod \"maas-controller-78cc949cd9-vpjbq\" (UID: \"771b760a-10af-45ad-9c18-673e4ae6e6f9\") " pod="opendatahub/maas-controller-78cc949cd9-vpjbq" Nov 02 17:37:36.702771 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.702738 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/771b760a-10af-45ad-9c18-673e4ae6e6f9-webhook-cert\") pod \"maas-controller-78cc949cd9-vpjbq\" (UID: \"771b760a-10af-45ad-9c18-673e4ae6e6f9\") " pod="opendatahub/maas-controller-78cc949cd9-vpjbq" Nov 02 17:37:36.702771 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.702774 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/771b760a-10af-45ad-9c18-673e4ae6e6f9-metrics-certs\") pod \"maas-controller-78cc949cd9-vpjbq\" (UID: \"771b760a-10af-45ad-9c18-673e4ae6e6f9\") " pod="opendatahub/maas-controller-78cc949cd9-vpjbq" Nov 02 17:37:36.702983 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.702795 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7flm\" (UniqueName: \"kubernetes.io/projected/771b760a-10af-45ad-9c18-673e4ae6e6f9-kube-api-access-v7flm\") pod \"maas-controller-78cc949cd9-vpjbq\" (UID: \"771b760a-10af-45ad-9c18-673e4ae6e6f9\") " pod="opendatahub/maas-controller-78cc949cd9-vpjbq" Nov 02 17:37:36.705204 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.705175 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/771b760a-10af-45ad-9c18-673e4ae6e6f9-metrics-certs\") pod \"maas-controller-78cc949cd9-vpjbq\" (UID: \"771b760a-10af-45ad-9c18-673e4ae6e6f9\") " pod="opendatahub/maas-controller-78cc949cd9-vpjbq" Nov 02 17:37:36.705322 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.705206 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/771b760a-10af-45ad-9c18-673e4ae6e6f9-webhook-cert\") pod \"maas-controller-78cc949cd9-vpjbq\" (UID: \"771b760a-10af-45ad-9c18-673e4ae6e6f9\") " pod="opendatahub/maas-controller-78cc949cd9-vpjbq" Nov 02 17:37:36.712165 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.712135 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7flm\" (UniqueName: \"kubernetes.io/projected/771b760a-10af-45ad-9c18-673e4ae6e6f9-kube-api-access-v7flm\") pod \"maas-controller-78cc949cd9-vpjbq\" (UID: \"771b760a-10af-45ad-9c18-673e4ae6e6f9\") " pod="opendatahub/maas-controller-78cc949cd9-vpjbq" Nov 02 17:37:36.809396 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.809309 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78cc949cd9-vpjbq" Nov 02 17:37:36.953286 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.953253 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56bc685b8f-572rh"] Nov 02 17:37:36.956450 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.956431 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bc685b8f-572rh" Nov 02 17:37:36.969541 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:36.969517 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56bc685b8f-572rh"] Nov 02 17:37:37.005254 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:37.005229 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-metrics-certs\") pod \"maas-controller-56bc685b8f-572rh\" (UID: \"2ff5f84a-63fe-4f69-bb19-ede4145e82f1\") " pod="opendatahub/maas-controller-56bc685b8f-572rh" Nov 02 17:37:37.005396 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:37.005259 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-webhook-cert\") pod \"maas-controller-56bc685b8f-572rh\" (UID: \"2ff5f84a-63fe-4f69-bb19-ede4145e82f1\") " pod="opendatahub/maas-controller-56bc685b8f-572rh" Nov 02 17:37:37.005396 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:37.005292 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkzkg\" (UniqueName: \"kubernetes.io/projected/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-kube-api-access-qkzkg\") pod \"maas-controller-56bc685b8f-572rh\" (UID: \"2ff5f84a-63fe-4f69-bb19-ede4145e82f1\") " pod="opendatahub/maas-controller-56bc685b8f-572rh" Nov 02 17:37:37.105948 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:37.105891 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-webhook-cert\") pod \"maas-controller-56bc685b8f-572rh\" (UID: \"2ff5f84a-63fe-4f69-bb19-ede4145e82f1\") " pod="opendatahub/maas-controller-56bc685b8f-572rh" Nov 02 17:37:37.105948 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:37.105942 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkzkg\" (UniqueName: \"kubernetes.io/projected/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-kube-api-access-qkzkg\") pod \"maas-controller-56bc685b8f-572rh\" (UID: \"2ff5f84a-63fe-4f69-bb19-ede4145e82f1\") " pod="opendatahub/maas-controller-56bc685b8f-572rh" Nov 02 17:37:37.106112 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:37.106030 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-metrics-certs\") pod \"maas-controller-56bc685b8f-572rh\" (UID: \"2ff5f84a-63fe-4f69-bb19-ede4145e82f1\") " pod="opendatahub/maas-controller-56bc685b8f-572rh" Nov 02 17:37:37.108309 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:37.108288 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-webhook-cert\") pod \"maas-controller-56bc685b8f-572rh\" (UID: \"2ff5f84a-63fe-4f69-bb19-ede4145e82f1\") " pod="opendatahub/maas-controller-56bc685b8f-572rh" Nov 02 17:37:37.108393 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:37.108360 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-metrics-certs\") pod \"maas-controller-56bc685b8f-572rh\" (UID: \"2ff5f84a-63fe-4f69-bb19-ede4145e82f1\") " pod="opendatahub/maas-controller-56bc685b8f-572rh" Nov 02 17:37:37.127387 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:37.127365 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78cc949cd9-vpjbq"] Nov 02 17:37:37.128846 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:37:37.128822 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771b760a_10af_45ad_9c18_673e4ae6e6f9.slice/crio-a5fea45b8666221bc25f6e9f41998beafcde8c0a2dca459886ffd4adf575c975 WatchSource:0}: Error finding container a5fea45b8666221bc25f6e9f41998beafcde8c0a2dca459886ffd4adf575c975: Status 404 returned error can't find the container with id a5fea45b8666221bc25f6e9f41998beafcde8c0a2dca459886ffd4adf575c975 Nov 02 17:37:37.129601 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:37.129582 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkzkg\" (UniqueName: \"kubernetes.io/projected/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-kube-api-access-qkzkg\") pod \"maas-controller-56bc685b8f-572rh\" (UID: \"2ff5f84a-63fe-4f69-bb19-ede4145e82f1\") " pod="opendatahub/maas-controller-56bc685b8f-572rh" Nov 02 17:37:37.268326 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:37.268286 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bc685b8f-572rh" Nov 02 17:37:37.545122 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:37.545099 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56bc685b8f-572rh"] Nov 02 17:37:37.547410 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:37:37.547386 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff5f84a_63fe_4f69_bb19_ede4145e82f1.slice/crio-8f9c3a42f5d411a6ed140a87a60854218c34f8f2ff709307c1a4eb2c354391d0 WatchSource:0}: Error finding container 8f9c3a42f5d411a6ed140a87a60854218c34f8f2ff709307c1a4eb2c354391d0: Status 404 returned error can't find the container with id 8f9c3a42f5d411a6ed140a87a60854218c34f8f2ff709307c1a4eb2c354391d0 Nov 02 17:37:37.649413 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:37.649379 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bc685b8f-572rh" event={"ID":"2ff5f84a-63fe-4f69-bb19-ede4145e82f1","Type":"ContainerStarted","Data":"8f9c3a42f5d411a6ed140a87a60854218c34f8f2ff709307c1a4eb2c354391d0"} Nov 02 17:37:37.650479 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:37.650453 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78cc949cd9-vpjbq" event={"ID":"771b760a-10af-45ad-9c18-673e4ae6e6f9","Type":"ContainerStarted","Data":"a5fea45b8666221bc25f6e9f41998beafcde8c0a2dca459886ffd4adf575c975"} Nov 02 17:37:41.674801 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:41.674701 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bc685b8f-572rh" event={"ID":"2ff5f84a-63fe-4f69-bb19-ede4145e82f1","Type":"ContainerStarted","Data":"44c7c81b02bb5e835b96ba4a88ce0c94d27ad41accbf394a3826891478d072a1"} Nov 02 17:37:41.674801 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:41.674770 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56bc685b8f-572rh" Nov 02 17:37:41.676602 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:41.676563 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78cc949cd9-vpjbq" event={"ID":"771b760a-10af-45ad-9c18-673e4ae6e6f9","Type":"ContainerStarted","Data":"30f83d338281d391ff6346c411b8f6cfa531b2a66328add527b1b01558a1aeec"} Nov 02 17:37:41.676729 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:41.676661 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78cc949cd9-vpjbq" Nov 02 17:37:41.694870 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:41.694807 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-56bc685b8f-572rh" podStartSLOduration=1.642792203 podStartE2EDuration="5.694789813s" totalImagesPullingTime="4.05199761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:41.693997377 +0000 UTC m=+226.385051387" watchObservedRunningTime="2026-11-02 17:37:41.694789813 +0000 UTC m=+226.385843828" Nov 02 17:37:41.715313 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:41.715261 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-78cc949cd9-vpjbq" podStartSLOduration=1.235493626 podStartE2EDuration="5.715247659s" totalImagesPullingTime="4.479754033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:41.714457783 +0000 UTC m=+226.405511796" watchObservedRunningTime="2026-11-02 17:37:41.715247659 +0000 UTC m=+226.406301668" Nov 02 17:37:49.247434 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.247385 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-t5dg5"] Nov 02 17:37:49.251083 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.251059 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-t5dg5" Nov 02 17:37:49.266932 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.265597 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-t5dg5"] Nov 02 17:37:49.273534 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.273482 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-w9k7w\"" Nov 02 17:37:49.273716 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.273607 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 02 17:37:49.275926 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.275901 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr"] Nov 02 17:37:49.280105 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.280083 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr" Nov 02 17:37:49.289231 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.289187 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-kg8z2\"" Nov 02 17:37:49.289349 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.289279 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 02 17:37:49.289881 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.289825 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr"] Nov 02 17:37:49.323549 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.323518 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aad4749c-18d8-4003-8576-211672ff42e5-cert\") pod \"kserve-controller-manager-776cc66d47-t5dg5\" (UID: \"aad4749c-18d8-4003-8576-211672ff42e5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-t5dg5" Nov 02 17:37:49.323726 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.323661 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8jh2\" (UniqueName: \"kubernetes.io/projected/aad4749c-18d8-4003-8576-211672ff42e5-kube-api-access-v8jh2\") pod \"kserve-controller-manager-776cc66d47-t5dg5\" (UID: \"aad4749c-18d8-4003-8576-211672ff42e5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-t5dg5" Nov 02 17:37:49.424399 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.424369 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8jh2\" (UniqueName: \"kubernetes.io/projected/aad4749c-18d8-4003-8576-211672ff42e5-kube-api-access-v8jh2\") pod \"kserve-controller-manager-776cc66d47-t5dg5\" (UID: \"aad4749c-18d8-4003-8576-211672ff42e5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-t5dg5" Nov 02 17:37:49.424561 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.424429 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2fnc\" (UniqueName: \"kubernetes.io/projected/60ede993-569b-4c15-8927-bf08bd194107-kube-api-access-q2fnc\") pod \"llmisvc-controller-manager-869d5dcc5d-lxmdr\" (UID: \"60ede993-569b-4c15-8927-bf08bd194107\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr" Nov 02 17:37:49.424561 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.424456 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aad4749c-18d8-4003-8576-211672ff42e5-cert\") pod \"kserve-controller-manager-776cc66d47-t5dg5\" (UID: \"aad4749c-18d8-4003-8576-211672ff42e5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-t5dg5" Nov 02 17:37:49.424561 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.424486 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60ede993-569b-4c15-8927-bf08bd194107-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-lxmdr\" (UID: \"60ede993-569b-4c15-8927-bf08bd194107\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr" Nov 02 17:37:49.426847 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.426820 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aad4749c-18d8-4003-8576-211672ff42e5-cert\") pod \"kserve-controller-manager-776cc66d47-t5dg5\" (UID: \"aad4749c-18d8-4003-8576-211672ff42e5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-t5dg5" Nov 02 17:37:49.437215 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.437187 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8jh2\" (UniqueName: \"kubernetes.io/projected/aad4749c-18d8-4003-8576-211672ff42e5-kube-api-access-v8jh2\") pod \"kserve-controller-manager-776cc66d47-t5dg5\" (UID: \"aad4749c-18d8-4003-8576-211672ff42e5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-t5dg5" Nov 02 17:37:49.525382 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.525294 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2fnc\" (UniqueName: \"kubernetes.io/projected/60ede993-569b-4c15-8927-bf08bd194107-kube-api-access-q2fnc\") pod \"llmisvc-controller-manager-869d5dcc5d-lxmdr\" (UID: \"60ede993-569b-4c15-8927-bf08bd194107\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr" Nov 02 17:37:49.525382 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.525366 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60ede993-569b-4c15-8927-bf08bd194107-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-lxmdr\" (UID: \"60ede993-569b-4c15-8927-bf08bd194107\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr" Nov 02 17:37:49.527832 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.527810 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60ede993-569b-4c15-8927-bf08bd194107-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-lxmdr\" (UID: \"60ede993-569b-4c15-8927-bf08bd194107\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr" Nov 02 17:37:49.533693 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.533670 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2fnc\" (UniqueName: \"kubernetes.io/projected/60ede993-569b-4c15-8927-bf08bd194107-kube-api-access-q2fnc\") pod \"llmisvc-controller-manager-869d5dcc5d-lxmdr\" (UID: \"60ede993-569b-4c15-8927-bf08bd194107\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr" Nov 02 17:37:49.572569 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.572536 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-t5dg5" Nov 02 17:37:49.595452 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.595420 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr" Nov 02 17:37:49.851717 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.851695 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-t5dg5"] Nov 02 17:37:49.854005 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:37:49.853973 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad4749c_18d8_4003_8576_211672ff42e5.slice/crio-dc31d37b2a155652b5b314bde9e4948767aac22cc6a4cc33f3fb4fcd8b331d00 WatchSource:0}: Error finding container dc31d37b2a155652b5b314bde9e4948767aac22cc6a4cc33f3fb4fcd8b331d00: Status 404 returned error can't find the container with id dc31d37b2a155652b5b314bde9e4948767aac22cc6a4cc33f3fb4fcd8b331d00 Nov 02 17:37:49.883687 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:49.883667 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr"] Nov 02 17:37:49.885471 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:37:49.885440 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod60ede993_569b_4c15_8927_bf08bd194107.slice/crio-fcec384d633d755672d85a9d7de34209f9296cc14436c1ed80f9cfd89e0f50c3 WatchSource:0}: Error finding container fcec384d633d755672d85a9d7de34209f9296cc14436c1ed80f9cfd89e0f50c3: Status 404 returned error can't find the container with id fcec384d633d755672d85a9d7de34209f9296cc14436c1ed80f9cfd89e0f50c3 Nov 02 17:37:50.721483 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:50.721441 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-t5dg5" event={"ID":"aad4749c-18d8-4003-8576-211672ff42e5","Type":"ContainerStarted","Data":"dc31d37b2a155652b5b314bde9e4948767aac22cc6a4cc33f3fb4fcd8b331d00"} Nov 02 17:37:50.723002 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:50.722965 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr" event={"ID":"60ede993-569b-4c15-8927-bf08bd194107","Type":"ContainerStarted","Data":"fcec384d633d755672d85a9d7de34209f9296cc14436c1ed80f9cfd89e0f50c3"} Nov 02 17:37:52.683752 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:52.683724 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56bc685b8f-572rh" Nov 02 17:37:52.684169 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:52.683793 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78cc949cd9-vpjbq" Nov 02 17:37:52.739546 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:52.739503 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr" event={"ID":"60ede993-569b-4c15-8927-bf08bd194107","Type":"ContainerStarted","Data":"069a86347fc66f1b0e09e4440083cbb7f6d322d499e84deb6a1c9d6d0f501808"} Nov 02 17:37:52.739726 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:52.739652 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr" Nov 02 17:37:52.741600 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:52.741567 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-t5dg5" event={"ID":"aad4749c-18d8-4003-8576-211672ff42e5","Type":"ContainerStarted","Data":"d16830bce0851255d166dff99d3fadecee5bb54433e88a010a76093b3684faed"} Nov 02 17:37:52.741927 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:52.741898 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-t5dg5" Nov 02 17:37:52.754766 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:52.754738 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78cc949cd9-vpjbq"] Nov 02 17:37:52.755246 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:52.755182 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-78cc949cd9-vpjbq" podUID="771b760a-10af-45ad-9c18-673e4ae6e6f9" containerName="manager" containerID="cri-o://30f83d338281d391ff6346c411b8f6cfa531b2a66328add527b1b01558a1aeec" gracePeriod=30 Nov 02 17:37:52.763674 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:52.763592 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr" podStartSLOduration=1.6392940249999999 podStartE2EDuration="3.763579945s" totalImagesPullingTime="2.12428592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:52.760706051 +0000 UTC m=+237.451760050" watchObservedRunningTime="2026-11-02 17:37:52.763579945 +0000 UTC m=+237.454633955" Nov 02 17:37:52.784026 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:52.783975 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-t5dg5" podStartSLOduration=1.631162899 podStartE2EDuration="3.78396229s" totalImagesPullingTime="2.152799391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:52.782936037 +0000 UTC m=+237.473990049" watchObservedRunningTime="2026-11-02 17:37:52.78396229 +0000 UTC m=+237.475016295" Nov 02 17:37:53.003735 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:53.003711 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78cc949cd9-vpjbq" Nov 02 17:37:53.170064 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:53.170033 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/771b760a-10af-45ad-9c18-673e4ae6e6f9-kube-api-access-v7flm\" (UniqueName: \"kubernetes.io/projected/771b760a-10af-45ad-9c18-673e4ae6e6f9-kube-api-access-v7flm\") pod \"771b760a-10af-45ad-9c18-673e4ae6e6f9\" (UID: \"771b760a-10af-45ad-9c18-673e4ae6e6f9\") " Nov 02 17:37:53.170261 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:53.170192 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/771b760a-10af-45ad-9c18-673e4ae6e6f9-webhook-cert\" (UniqueName: \"kubernetes.io/secret/771b760a-10af-45ad-9c18-673e4ae6e6f9-webhook-cert\") pod \"771b760a-10af-45ad-9c18-673e4ae6e6f9\" (UID: \"771b760a-10af-45ad-9c18-673e4ae6e6f9\") " Nov 02 17:37:53.170261 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:53.170225 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/771b760a-10af-45ad-9c18-673e4ae6e6f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/771b760a-10af-45ad-9c18-673e4ae6e6f9-metrics-certs\") pod \"771b760a-10af-45ad-9c18-673e4ae6e6f9\" (UID: \"771b760a-10af-45ad-9c18-673e4ae6e6f9\") " Nov 02 17:37:53.172337 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:53.172300 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/771b760a-10af-45ad-9c18-673e4ae6e6f9-metrics-certs" pod "771b760a-10af-45ad-9c18-673e4ae6e6f9" (UID: "771b760a-10af-45ad-9c18-673e4ae6e6f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:37:53.172337 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:53.172320 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/771b760a-10af-45ad-9c18-673e4ae6e6f9-webhook-cert" pod "771b760a-10af-45ad-9c18-673e4ae6e6f9" (UID: "771b760a-10af-45ad-9c18-673e4ae6e6f9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:37:53.172487 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:53.172310 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771b760a-10af-45ad-9c18-673e4ae6e6f9-kube-api-access-v7flm" pod "771b760a-10af-45ad-9c18-673e4ae6e6f9" (UID: "771b760a-10af-45ad-9c18-673e4ae6e6f9"). InnerVolumeSpecName "kube-api-access-v7flm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:37:53.271656 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:53.271556 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7flm\" (UniqueName: \"kubernetes.io/projected/771b760a-10af-45ad-9c18-673e4ae6e6f9-kube-api-access-v7flm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:37:53.271656 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:53.271586 5601 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/771b760a-10af-45ad-9c18-673e4ae6e6f9-webhook-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:37:53.271656 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:53.271596 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/771b760a-10af-45ad-9c18-673e4ae6e6f9-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:37:53.748176 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:53.748142 5601 generic.go:356] "Generic (PLEG): container finished" podID="771b760a-10af-45ad-9c18-673e4ae6e6f9" containerID="30f83d338281d391ff6346c411b8f6cfa531b2a66328add527b1b01558a1aeec" exitCode=0 Nov 02 17:37:53.748600 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:53.748197 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78cc949cd9-vpjbq" Nov 02 17:37:53.748600 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:53.748228 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78cc949cd9-vpjbq" event={"ID":"771b760a-10af-45ad-9c18-673e4ae6e6f9","Type":"ContainerDied","Data":"30f83d338281d391ff6346c411b8f6cfa531b2a66328add527b1b01558a1aeec"} Nov 02 17:37:53.748600 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:53.748282 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78cc949cd9-vpjbq" event={"ID":"771b760a-10af-45ad-9c18-673e4ae6e6f9","Type":"ContainerDied","Data":"a5fea45b8666221bc25f6e9f41998beafcde8c0a2dca459886ffd4adf575c975"} Nov 02 17:37:53.748600 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:53.748313 5601 scope.go:122] "RemoveContainer" containerID="30f83d338281d391ff6346c411b8f6cfa531b2a66328add527b1b01558a1aeec" Nov 02 17:37:53.756756 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:53.756687 5601 scope.go:122] "RemoveContainer" containerID="30f83d338281d391ff6346c411b8f6cfa531b2a66328add527b1b01558a1aeec" Nov 02 17:37:53.757003 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:37:53.756981 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f83d338281d391ff6346c411b8f6cfa531b2a66328add527b1b01558a1aeec\": container with ID starting with 30f83d338281d391ff6346c411b8f6cfa531b2a66328add527b1b01558a1aeec not found: ID does not exist" containerID="30f83d338281d391ff6346c411b8f6cfa531b2a66328add527b1b01558a1aeec" Nov 02 17:37:53.757071 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:53.757015 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f83d338281d391ff6346c411b8f6cfa531b2a66328add527b1b01558a1aeec"} err="failed to get container status \"30f83d338281d391ff6346c411b8f6cfa531b2a66328add527b1b01558a1aeec\": rpc error: code = NotFound desc = could not find container \"30f83d338281d391ff6346c411b8f6cfa531b2a66328add527b1b01558a1aeec\": container with ID starting with 30f83d338281d391ff6346c411b8f6cfa531b2a66328add527b1b01558a1aeec not found: ID does not exist" Nov 02 17:37:53.774341 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:53.774317 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78cc949cd9-vpjbq"] Nov 02 17:37:53.782717 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:53.782695 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78cc949cd9-vpjbq"] Nov 02 17:37:55.424913 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:37:55.424874 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="771b760a-10af-45ad-9c18-673e4ae6e6f9" path="/var/lib/kubelet/pods/771b760a-10af-45ad-9c18-673e4ae6e6f9/volumes" Nov 02 17:38:04.483485 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:38:04.483449 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:38:14.575605 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.575564 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq"] Nov 02 17:38:14.576072 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.576053 5601 cpu_manager.go:414] "containerMap: removing container" podUID="771b760a-10af-45ad-9c18-673e4ae6e6f9" containerName="manager" Nov 02 17:38:14.576145 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.576075 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="771b760a-10af-45ad-9c18-673e4ae6e6f9" containerName="manager" Nov 02 17:38:14.576189 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.576176 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="771b760a-10af-45ad-9c18-673e4ae6e6f9" containerName="manager" Nov 02 17:38:14.580782 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.580759 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" Nov 02 17:38:14.587864 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.587828 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tqgzn\"" Nov 02 17:38:14.589475 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.589451 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 17:38:14.589684 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.589560 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 17:38:14.590291 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.590269 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq"] Nov 02 17:38:14.656702 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.656668 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-openshift-service-ca\") pod \"maas-api-75b6bd7596-cfdjq\" (UID: \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\") " pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" Nov 02 17:38:14.656919 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.656733 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-metrics-certs\") pod \"maas-api-75b6bd7596-cfdjq\" (UID: \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\") " pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" Nov 02 17:38:14.656919 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.656763 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkl64\" (UniqueName: \"kubernetes.io/projected/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-kube-api-access-pkl64\") pod \"maas-api-75b6bd7596-cfdjq\" (UID: \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\") " pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" Nov 02 17:38:14.657156 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.657125 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-maas-api-tls\") pod \"maas-api-75b6bd7596-cfdjq\" (UID: \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\") " pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" Nov 02 17:38:14.758814 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.758772 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-metrics-certs\") pod \"maas-api-75b6bd7596-cfdjq\" (UID: \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\") " pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" Nov 02 17:38:14.759015 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.758824 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkl64\" (UniqueName: \"kubernetes.io/projected/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-kube-api-access-pkl64\") pod \"maas-api-75b6bd7596-cfdjq\" (UID: \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\") " pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" Nov 02 17:38:14.759015 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.758884 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-maas-api-tls\") pod \"maas-api-75b6bd7596-cfdjq\" (UID: \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\") " pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" Nov 02 17:38:14.759015 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.758975 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-openshift-service-ca\") pod \"maas-api-75b6bd7596-cfdjq\" (UID: \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\") " pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" Nov 02 17:38:14.759699 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.759677 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-openshift-service-ca\") pod \"maas-api-75b6bd7596-cfdjq\" (UID: \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\") " pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" Nov 02 17:38:14.761397 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.761366 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-metrics-certs\") pod \"maas-api-75b6bd7596-cfdjq\" (UID: \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\") " pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" Nov 02 17:38:14.761498 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.761427 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-maas-api-tls\") pod \"maas-api-75b6bd7596-cfdjq\" (UID: \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\") " pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" Nov 02 17:38:14.767045 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.767023 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkl64\" (UniqueName: \"kubernetes.io/projected/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-kube-api-access-pkl64\") pod \"maas-api-75b6bd7596-cfdjq\" (UID: \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\") " pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" Nov 02 17:38:14.895117 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:14.895030 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" Nov 02 17:38:15.175936 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:15.175819 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq"] Nov 02 17:38:15.178776 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:38:15.178743 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce0db90_cb53_4d00_bae2_f5d95b7bab5b.slice/crio-52c4937d161c531ec73b7e9f226a6872e7662495e5ef69a0ab6c5bb2f9a5c434 WatchSource:0}: Error finding container 52c4937d161c531ec73b7e9f226a6872e7662495e5ef69a0ab6c5bb2f9a5c434: Status 404 returned error can't find the container with id 52c4937d161c531ec73b7e9f226a6872e7662495e5ef69a0ab6c5bb2f9a5c434 Nov 02 17:38:15.866129 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:15.866096 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" event={"ID":"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b","Type":"ContainerStarted","Data":"52c4937d161c531ec73b7e9f226a6872e7662495e5ef69a0ab6c5bb2f9a5c434"} Nov 02 17:38:19.888705 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:19.888662 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" event={"ID":"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b","Type":"ContainerStarted","Data":"5790e002041125a84b20a1e73f35521a997a72a142ab3c8d1f60aabbc99e97cb"} Nov 02 17:38:19.889234 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:19.888770 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" Nov 02 17:38:19.907987 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:19.907922 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" podStartSLOduration=1.756886523 podStartE2EDuration="5.907907901s" totalImagesPullingTime="4.151021378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:19.906405879 +0000 UTC m=+264.597459904" watchObservedRunningTime="2026-11-02 17:38:19.907907901 +0000 UTC m=+264.598961912" Nov 02 17:38:23.750802 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:23.750764 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr" Nov 02 17:38:23.753733 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:23.753706 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-t5dg5" Nov 02 17:38:24.918296 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:24.918212 5601 generic.go:356] "Generic (PLEG): container finished" podID="63f70366-70f1-475a-8e45-eb6da6b4828e" containerID="ef9d8fd509c7278e026a2048271a935ccca8262d5ce15ad0c2bc2896aa6eaa6d" exitCode=0 Nov 02 17:38:24.918296 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:24.918271 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" event={"ID":"63f70366-70f1-475a-8e45-eb6da6b4828e","Type":"ContainerDied","Data":"ef9d8fd509c7278e026a2048271a935ccca8262d5ce15ad0c2bc2896aa6eaa6d"} Nov 02 17:38:24.918804 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:24.918640 5601 scope.go:122] "RemoveContainer" containerID="ef9d8fd509c7278e026a2048271a935ccca8262d5ce15ad0c2bc2896aa6eaa6d" Nov 02 17:38:24.919808 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:24.919780 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:38:24.919880 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:24.919846 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:38:24.919916 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:24.919884 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:38:25.900353 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:25.900320 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" Nov 02 17:38:25.926020 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:25.925985 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" event={"ID":"63f70366-70f1-475a-8e45-eb6da6b4828e","Type":"ContainerStarted","Data":"fbfaa5612f264b511dbaa7fb11360a90079f670f676ec11bbee257262383869b"} Nov 02 17:38:26.649639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:26.636055 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kgz84"] Nov 02 17:38:26.660899 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:26.660868 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kgz84"] Nov 02 17:38:26.661079 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:26.660958 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-kgz84" Nov 02 17:38:26.671754 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:26.671726 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f245n\" (UniqueName: \"kubernetes.io/projected/47e80c19-ed47-435c-9b7d-5ca7ec6eb2bb-kube-api-access-f245n\") pod \"authorino-6f6944bcdd-kgz84\" (UID: \"47e80c19-ed47-435c-9b7d-5ca7ec6eb2bb\") " pod="kuadrant-system/authorino-6f6944bcdd-kgz84" Nov 02 17:38:26.755642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:26.736277 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kgz84"] Nov 02 17:38:26.756214 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:38:26.756173 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-f245n], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-kgz84" podUID="47e80c19-ed47-435c-9b7d-5ca7ec6eb2bb" Nov 02 17:38:26.773247 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:26.773214 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f245n\" (UniqueName: \"kubernetes.io/projected/47e80c19-ed47-435c-9b7d-5ca7ec6eb2bb-kube-api-access-f245n\") pod \"authorino-6f6944bcdd-kgz84\" (UID: \"47e80c19-ed47-435c-9b7d-5ca7ec6eb2bb\") " pod="kuadrant-system/authorino-6f6944bcdd-kgz84" Nov 02 17:38:26.786510 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:26.786471 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f245n\" (UniqueName: \"kubernetes.io/projected/47e80c19-ed47-435c-9b7d-5ca7ec6eb2bb-kube-api-access-f245n\") pod \"authorino-6f6944bcdd-kgz84\" (UID: \"47e80c19-ed47-435c-9b7d-5ca7ec6eb2bb\") " pod="kuadrant-system/authorino-6f6944bcdd-kgz84" Nov 02 17:38:26.914849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:26.914764 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:38:26.915994 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:26.915959 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-wtks5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body= Nov 02 17:38:26.916126 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:26.916017 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" podUID="63f70366-70f1-475a-8e45-eb6da6b4828e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" Nov 02 17:38:26.930658 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:26.930636 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-kgz84" Nov 02 17:38:26.962502 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:26.962477 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-kgz84" Nov 02 17:38:26.975280 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:26.975254 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e80c19-ed47-435c-9b7d-5ca7ec6eb2bb-kube-api-access-f245n\" (UniqueName: \"kubernetes.io/projected/47e80c19-ed47-435c-9b7d-5ca7ec6eb2bb-kube-api-access-f245n\") pod \"47e80c19-ed47-435c-9b7d-5ca7ec6eb2bb\" (UID: \"47e80c19-ed47-435c-9b7d-5ca7ec6eb2bb\") " Nov 02 17:38:26.977302 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:26.977272 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e80c19-ed47-435c-9b7d-5ca7ec6eb2bb-kube-api-access-f245n" pod "47e80c19-ed47-435c-9b7d-5ca7ec6eb2bb" (UID: "47e80c19-ed47-435c-9b7d-5ca7ec6eb2bb"). InnerVolumeSpecName "kube-api-access-f245n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:38:27.017592 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:27.017555 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68d65c5784-6mkqh"] Nov 02 17:38:27.020080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:27.020063 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68d65c5784-6mkqh" Nov 02 17:38:27.029200 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:27.029176 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 02 17:38:27.030154 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:27.030131 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68d65c5784-6mkqh"] Nov 02 17:38:27.076387 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:27.076342 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/821376c6-e692-4455-bfd0-d7d4158fa006-tls-cert\") pod \"authorino-68d65c5784-6mkqh\" (UID: \"821376c6-e692-4455-bfd0-d7d4158fa006\") " pod="kuadrant-system/authorino-68d65c5784-6mkqh" Nov 02 17:38:27.076591 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:27.076468 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrz5l\" (UniqueName: \"kubernetes.io/projected/821376c6-e692-4455-bfd0-d7d4158fa006-kube-api-access-rrz5l\") pod \"authorino-68d65c5784-6mkqh\" (UID: \"821376c6-e692-4455-bfd0-d7d4158fa006\") " pod="kuadrant-system/authorino-68d65c5784-6mkqh" Nov 02 17:38:27.076591 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:27.076531 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f245n\" (UniqueName: \"kubernetes.io/projected/47e80c19-ed47-435c-9b7d-5ca7ec6eb2bb-kube-api-access-f245n\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:38:27.177765 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:27.177731 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrz5l\" (UniqueName: \"kubernetes.io/projected/821376c6-e692-4455-bfd0-d7d4158fa006-kube-api-access-rrz5l\") pod \"authorino-68d65c5784-6mkqh\" (UID: \"821376c6-e692-4455-bfd0-d7d4158fa006\") " pod="kuadrant-system/authorino-68d65c5784-6mkqh" Nov 02 17:38:27.177923 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:27.177792 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/821376c6-e692-4455-bfd0-d7d4158fa006-tls-cert\") pod \"authorino-68d65c5784-6mkqh\" (UID: \"821376c6-e692-4455-bfd0-d7d4158fa006\") " pod="kuadrant-system/authorino-68d65c5784-6mkqh" Nov 02 17:38:27.180194 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:27.180165 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/821376c6-e692-4455-bfd0-d7d4158fa006-tls-cert\") pod \"authorino-68d65c5784-6mkqh\" (UID: \"821376c6-e692-4455-bfd0-d7d4158fa006\") " pod="kuadrant-system/authorino-68d65c5784-6mkqh" Nov 02 17:38:27.190759 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:27.190737 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrz5l\" (UniqueName: \"kubernetes.io/projected/821376c6-e692-4455-bfd0-d7d4158fa006-kube-api-access-rrz5l\") pod \"authorino-68d65c5784-6mkqh\" (UID: \"821376c6-e692-4455-bfd0-d7d4158fa006\") " pod="kuadrant-system/authorino-68d65c5784-6mkqh" Nov 02 17:38:27.332447 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:27.332408 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68d65c5784-6mkqh" Nov 02 17:38:27.623523 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:27.623489 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68d65c5784-6mkqh"] Nov 02 17:38:27.626639 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:38:27.626592 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod821376c6_e692_4455_bfd0_d7d4158fa006.slice/crio-89c969cd796365f7f920d50984379c9ba6b2ccdeaa553fff1f471c95d1f1e15e WatchSource:0}: Error finding container 89c969cd796365f7f920d50984379c9ba6b2ccdeaa553fff1f471c95d1f1e15e: Status 404 returned error can't find the container with id 89c969cd796365f7f920d50984379c9ba6b2ccdeaa553fff1f471c95d1f1e15e Nov 02 17:38:27.915400 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:27.915317 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-wtks5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body= Nov 02 17:38:27.915400 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:27.915373 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" podUID="63f70366-70f1-475a-8e45-eb6da6b4828e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" Nov 02 17:38:27.960747 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:27.960720 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-kgz84" Nov 02 17:38:27.961244 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:27.960720 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68d65c5784-6mkqh" event={"ID":"821376c6-e692-4455-bfd0-d7d4158fa006","Type":"ContainerStarted","Data":"89c969cd796365f7f920d50984379c9ba6b2ccdeaa553fff1f471c95d1f1e15e"} Nov 02 17:38:28.001790 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:28.001755 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kgz84"] Nov 02 17:38:28.009684 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:28.009653 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kgz84"] Nov 02 17:38:28.915775 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:28.915686 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-wtks5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body= Nov 02 17:38:28.915775 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:28.915759 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" podUID="63f70366-70f1-475a-8e45-eb6da6b4828e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" Nov 02 17:38:28.976803 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:28.976755 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68d65c5784-6mkqh" event={"ID":"821376c6-e692-4455-bfd0-d7d4158fa006","Type":"ContainerStarted","Data":"e570bc049a995c176bf536f454b8aa8ad7db6c046a82450a2e307f563ff2d3cc"} Nov 02 17:38:29.003515 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:29.003435 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-68d65c5784-6mkqh" podStartSLOduration=1.200954031 podStartE2EDuration="2.003414261s" totalImagesPullingTime="802.46023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:29.001087047 +0000 UTC m=+273.692141126" watchObservedRunningTime="2026-11-02 17:38:29.003414261 +0000 UTC m=+273.694468274" Nov 02 17:38:29.052640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:29.050831 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-d7z2j"] Nov 02 17:38:29.052640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:29.051083 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-d7z2j" podUID="f5865ba9-7951-4082-b7b0-743110225f59" containerName="authorino" containerID="cri-o://c4cd5fd106ee4759b4582d3cab2be9d1189d745d61fdeaaa7a49a4de129fbc40" gracePeriod=30 Nov 02 17:38:29.316894 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:29.316862 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-d7z2j" Nov 02 17:38:29.397401 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:29.397360 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5865ba9-7951-4082-b7b0-743110225f59-kube-api-access-psgqw\" (UniqueName: \"kubernetes.io/projected/f5865ba9-7951-4082-b7b0-743110225f59-kube-api-access-psgqw\") pod \"f5865ba9-7951-4082-b7b0-743110225f59\" (UID: \"f5865ba9-7951-4082-b7b0-743110225f59\") " Nov 02 17:38:29.399605 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:29.399579 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5865ba9-7951-4082-b7b0-743110225f59-kube-api-access-psgqw" pod "f5865ba9-7951-4082-b7b0-743110225f59" (UID: "f5865ba9-7951-4082-b7b0-743110225f59"). InnerVolumeSpecName "kube-api-access-psgqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:38:29.424132 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:29.424052 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e80c19-ed47-435c-9b7d-5ca7ec6eb2bb" path="/var/lib/kubelet/pods/47e80c19-ed47-435c-9b7d-5ca7ec6eb2bb/volumes" Nov 02 17:38:29.498573 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:29.498527 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psgqw\" (UniqueName: \"kubernetes.io/projected/f5865ba9-7951-4082-b7b0-743110225f59-kube-api-access-psgqw\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:38:29.915079 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:29.914997 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-wtks5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body= Nov 02 17:38:29.915079 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:29.915054 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" podUID="63f70366-70f1-475a-8e45-eb6da6b4828e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" Nov 02 17:38:29.982811 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:29.982773 5601 generic.go:356] "Generic (PLEG): container finished" podID="f5865ba9-7951-4082-b7b0-743110225f59" containerID="c4cd5fd106ee4759b4582d3cab2be9d1189d745d61fdeaaa7a49a4de129fbc40" exitCode=0 Nov 02 17:38:29.983368 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:29.982825 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-d7z2j" Nov 02 17:38:29.983368 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:29.982860 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-d7z2j" event={"ID":"f5865ba9-7951-4082-b7b0-743110225f59","Type":"ContainerDied","Data":"c4cd5fd106ee4759b4582d3cab2be9d1189d745d61fdeaaa7a49a4de129fbc40"} Nov 02 17:38:29.983368 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:29.982902 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-d7z2j" event={"ID":"f5865ba9-7951-4082-b7b0-743110225f59","Type":"ContainerDied","Data":"a0d91b1cd66876cbc5360e8e4cd98a661e51bdc004e0a09d38deab7c08a525a0"} Nov 02 17:38:29.983368 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:29.982920 5601 scope.go:122] "RemoveContainer" containerID="c4cd5fd106ee4759b4582d3cab2be9d1189d745d61fdeaaa7a49a4de129fbc40" Nov 02 17:38:29.991686 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:29.991667 5601 scope.go:122] "RemoveContainer" containerID="c4cd5fd106ee4759b4582d3cab2be9d1189d745d61fdeaaa7a49a4de129fbc40" Nov 02 17:38:29.991960 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:38:29.991939 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4cd5fd106ee4759b4582d3cab2be9d1189d745d61fdeaaa7a49a4de129fbc40\": container with ID starting with c4cd5fd106ee4759b4582d3cab2be9d1189d745d61fdeaaa7a49a4de129fbc40 not found: ID does not exist" containerID="c4cd5fd106ee4759b4582d3cab2be9d1189d745d61fdeaaa7a49a4de129fbc40" Nov 02 17:38:29.992021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:29.991970 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4cd5fd106ee4759b4582d3cab2be9d1189d745d61fdeaaa7a49a4de129fbc40"} err="failed to get container status \"c4cd5fd106ee4759b4582d3cab2be9d1189d745d61fdeaaa7a49a4de129fbc40\": rpc error: code = NotFound desc = could not find container \"c4cd5fd106ee4759b4582d3cab2be9d1189d745d61fdeaaa7a49a4de129fbc40\": container with ID starting with c4cd5fd106ee4759b4582d3cab2be9d1189d745d61fdeaaa7a49a4de129fbc40 not found: ID does not exist" Nov 02 17:38:30.003651 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:30.003606 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-d7z2j"] Nov 02 17:38:30.010815 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:30.010784 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-d7z2j"] Nov 02 17:38:30.952678 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:30.952646 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:38:30.953271 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:30.953245 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:38:30.954104 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:30.954080 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" Nov 02 17:38:31.424589 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:31.424555 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5865ba9-7951-4082-b7b0-743110225f59" path="/var/lib/kubelet/pods/f5865ba9-7951-4082-b7b0-743110225f59/volumes" Nov 02 17:38:34.483991 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:38:34.483937 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:38:37.532420 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:37.532348 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq"] Nov 02 17:38:37.532825 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:37.532590 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" podUID="9ce0db90-cb53-4d00-bae2-f5d95b7bab5b" containerName="maas-api" containerID="cri-o://5790e002041125a84b20a1e73f35521a997a72a142ab3c8d1f60aabbc99e97cb" gracePeriod=30 Nov 02 17:38:37.776001 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:37.775977 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" Nov 02 17:38:37.877876 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:37.877793 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-metrics-certs\") pod \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\" (UID: \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\") " Nov 02 17:38:37.877876 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:37.877836 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-kube-api-access-pkl64\" (UniqueName: \"kubernetes.io/projected/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-kube-api-access-pkl64\") pod \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\" (UID: \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\") " Nov 02 17:38:37.877876 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:37.877867 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-openshift-service-ca\") pod \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\" (UID: \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\") " Nov 02 17:38:37.878110 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:37.878013 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-maas-api-tls\") pod \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\" (UID: \"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b\") " Nov 02 17:38:37.878259 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:37.878230 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-openshift-service-ca" pod "9ce0db90-cb53-4d00-bae2-f5d95b7bab5b" (UID: "9ce0db90-cb53-4d00-bae2-f5d95b7bab5b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:38:37.880065 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:37.880031 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-maas-api-tls" pod "9ce0db90-cb53-4d00-bae2-f5d95b7bab5b" (UID: "9ce0db90-cb53-4d00-bae2-f5d95b7bab5b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:38:37.880172 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:37.880076 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-metrics-certs" pod "9ce0db90-cb53-4d00-bae2-f5d95b7bab5b" (UID: "9ce0db90-cb53-4d00-bae2-f5d95b7bab5b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:38:37.880172 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:37.880115 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-kube-api-access-pkl64" pod "9ce0db90-cb53-4d00-bae2-f5d95b7bab5b" (UID: "9ce0db90-cb53-4d00-bae2-f5d95b7bab5b"). InnerVolumeSpecName "kube-api-access-pkl64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:38:37.979498 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:37.979453 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:38:37.979498 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:37.979493 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:38:37.979498 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:37.979507 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkl64\" (UniqueName: \"kubernetes.io/projected/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-kube-api-access-pkl64\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:38:37.979728 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:37.979521 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:38:38.022792 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:38.022756 5601 generic.go:356] "Generic (PLEG): container finished" podID="9ce0db90-cb53-4d00-bae2-f5d95b7bab5b" containerID="5790e002041125a84b20a1e73f35521a997a72a142ab3c8d1f60aabbc99e97cb" exitCode=0 Nov 02 17:38:38.022927 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:38.022803 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" event={"ID":"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b","Type":"ContainerDied","Data":"5790e002041125a84b20a1e73f35521a997a72a142ab3c8d1f60aabbc99e97cb"} Nov 02 17:38:38.022927 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:38.022810 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" Nov 02 17:38:38.022927 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:38.022843 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq" event={"ID":"9ce0db90-cb53-4d00-bae2-f5d95b7bab5b","Type":"ContainerDied","Data":"52c4937d161c531ec73b7e9f226a6872e7662495e5ef69a0ab6c5bb2f9a5c434"} Nov 02 17:38:38.022927 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:38.022865 5601 scope.go:122] "RemoveContainer" containerID="5790e002041125a84b20a1e73f35521a997a72a142ab3c8d1f60aabbc99e97cb" Nov 02 17:38:38.032389 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:38.032369 5601 scope.go:122] "RemoveContainer" containerID="5790e002041125a84b20a1e73f35521a997a72a142ab3c8d1f60aabbc99e97cb" Nov 02 17:38:38.032747 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:38:38.032728 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5790e002041125a84b20a1e73f35521a997a72a142ab3c8d1f60aabbc99e97cb\": container with ID starting with 5790e002041125a84b20a1e73f35521a997a72a142ab3c8d1f60aabbc99e97cb not found: ID does not exist" containerID="5790e002041125a84b20a1e73f35521a997a72a142ab3c8d1f60aabbc99e97cb" Nov 02 17:38:38.032800 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:38.032757 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5790e002041125a84b20a1e73f35521a997a72a142ab3c8d1f60aabbc99e97cb"} err="failed to get container status \"5790e002041125a84b20a1e73f35521a997a72a142ab3c8d1f60aabbc99e97cb\": rpc error: code = NotFound desc = could not find container \"5790e002041125a84b20a1e73f35521a997a72a142ab3c8d1f60aabbc99e97cb\": container with ID starting with 5790e002041125a84b20a1e73f35521a997a72a142ab3c8d1f60aabbc99e97cb not found: ID does not exist" Nov 02 17:38:38.045664 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:38.045637 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq"] Nov 02 17:38:38.050189 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:38.050168 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-75b6bd7596-cfdjq"] Nov 02 17:38:39.423973 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:39.423935 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ce0db90-cb53-4d00-bae2-f5d95b7bab5b" path="/var/lib/kubelet/pods/9ce0db90-cb53-4d00-bae2-f5d95b7bab5b/volumes" Nov 02 17:38:50.131372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:50.131338 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-hsptq"] Nov 02 17:38:50.131831 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:50.131700 5601 cpu_manager.go:414] "containerMap: removing container" podUID="9ce0db90-cb53-4d00-bae2-f5d95b7bab5b" containerName="maas-api" Nov 02 17:38:50.131831 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:50.131712 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce0db90-cb53-4d00-bae2-f5d95b7bab5b" containerName="maas-api" Nov 02 17:38:50.131831 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:50.131750 5601 cpu_manager.go:414] "containerMap: removing container" podUID="f5865ba9-7951-4082-b7b0-743110225f59" containerName="authorino" Nov 02 17:38:50.131831 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:50.131755 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5865ba9-7951-4082-b7b0-743110225f59" containerName="authorino" Nov 02 17:38:50.131831 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:50.131812 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5865ba9-7951-4082-b7b0-743110225f59" containerName="authorino" Nov 02 17:38:50.131831 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:50.131820 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ce0db90-cb53-4d00-bae2-f5d95b7bab5b" containerName="maas-api" Nov 02 17:38:50.136305 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:50.136287 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hsptq" Nov 02 17:38:50.143641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:50.143405 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-hsptq"] Nov 02 17:38:50.291543 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:50.291507 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm2xj\" (UniqueName: \"kubernetes.io/projected/3aa1eac9-b956-4608-8c5a-343b37a69d70-kube-api-access-mm2xj\") pod \"llmisvc-controller-manager-6466c554d8-hsptq\" (UID: \"3aa1eac9-b956-4608-8c5a-343b37a69d70\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hsptq" Nov 02 17:38:50.291748 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:50.291666 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3aa1eac9-b956-4608-8c5a-343b37a69d70-cert\") pod \"llmisvc-controller-manager-6466c554d8-hsptq\" (UID: \"3aa1eac9-b956-4608-8c5a-343b37a69d70\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hsptq" Nov 02 17:38:50.392374 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:50.392268 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3aa1eac9-b956-4608-8c5a-343b37a69d70-cert\") pod \"llmisvc-controller-manager-6466c554d8-hsptq\" (UID: \"3aa1eac9-b956-4608-8c5a-343b37a69d70\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hsptq" Nov 02 17:38:50.392374 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:50.392331 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm2xj\" (UniqueName: \"kubernetes.io/projected/3aa1eac9-b956-4608-8c5a-343b37a69d70-kube-api-access-mm2xj\") pod \"llmisvc-controller-manager-6466c554d8-hsptq\" (UID: \"3aa1eac9-b956-4608-8c5a-343b37a69d70\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hsptq" Nov 02 17:38:50.394863 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:50.394837 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3aa1eac9-b956-4608-8c5a-343b37a69d70-cert\") pod \"llmisvc-controller-manager-6466c554d8-hsptq\" (UID: \"3aa1eac9-b956-4608-8c5a-343b37a69d70\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hsptq" Nov 02 17:38:50.402796 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:50.402766 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm2xj\" (UniqueName: \"kubernetes.io/projected/3aa1eac9-b956-4608-8c5a-343b37a69d70-kube-api-access-mm2xj\") pod \"llmisvc-controller-manager-6466c554d8-hsptq\" (UID: \"3aa1eac9-b956-4608-8c5a-343b37a69d70\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hsptq" Nov 02 17:38:50.450794 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:50.450756 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hsptq" Nov 02 17:38:50.725505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:50.725482 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-hsptq"] Nov 02 17:38:50.727551 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:38:50.727517 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa1eac9_b956_4608_8c5a_343b37a69d70.slice/crio-c874e2ef0a6d02407670b3067a21f280311831c1be3682fe160ab92089c0e7d8 WatchSource:0}: Error finding container c874e2ef0a6d02407670b3067a21f280311831c1be3682fe160ab92089c0e7d8: Status 404 returned error can't find the container with id c874e2ef0a6d02407670b3067a21f280311831c1be3682fe160ab92089c0e7d8 Nov 02 17:38:51.085832 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:51.085806 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hsptq" event={"ID":"3aa1eac9-b956-4608-8c5a-343b37a69d70","Type":"ContainerStarted","Data":"c874e2ef0a6d02407670b3067a21f280311831c1be3682fe160ab92089c0e7d8"} Nov 02 17:38:52.092007 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:52.091967 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hsptq" event={"ID":"3aa1eac9-b956-4608-8c5a-343b37a69d70","Type":"ContainerStarted","Data":"5cb4f525b382af0c8774672a76909dc44b39f08c3c2ce06794a24e62ac677b46"} Nov 02 17:38:52.092396 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:52.092121 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hsptq" Nov 02 17:38:52.111131 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:38:52.111083 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hsptq" podStartSLOduration=1.7541139449999998 podStartE2EDuration="2.111070981s" totalImagesPullingTime="356.957036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:52.109328852 +0000 UTC m=+296.800382893" watchObservedRunningTime="2026-11-02 17:38:52.111070981 +0000 UTC m=+296.802124992" Nov 02 17:39:04.483930 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:39:04.483891 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:39:23.099552 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:23.099518 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hsptq" Nov 02 17:39:23.145755 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:23.145724 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr"] Nov 02 17:39:23.146111 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:23.146046 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr" podUID="60ede993-569b-4c15-8927-bf08bd194107" containerName="manager" containerID="cri-o://069a86347fc66f1b0e09e4440083cbb7f6d322d499e84deb6a1c9d6d0f501808" gracePeriod=30 Nov 02 17:39:23.387672 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:23.387645 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr" Nov 02 17:39:23.478140 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:23.478110 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60ede993-569b-4c15-8927-bf08bd194107-cert\" (UniqueName: \"kubernetes.io/secret/60ede993-569b-4c15-8927-bf08bd194107-cert\") pod \"60ede993-569b-4c15-8927-bf08bd194107\" (UID: \"60ede993-569b-4c15-8927-bf08bd194107\") " Nov 02 17:39:23.478301 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:23.478170 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ede993-569b-4c15-8927-bf08bd194107-kube-api-access-q2fnc\" (UniqueName: \"kubernetes.io/projected/60ede993-569b-4c15-8927-bf08bd194107-kube-api-access-q2fnc\") pod \"60ede993-569b-4c15-8927-bf08bd194107\" (UID: \"60ede993-569b-4c15-8927-bf08bd194107\") " Nov 02 17:39:23.480163 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:23.480131 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ede993-569b-4c15-8927-bf08bd194107-cert" pod "60ede993-569b-4c15-8927-bf08bd194107" (UID: "60ede993-569b-4c15-8927-bf08bd194107"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:39:23.480301 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:23.480185 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ede993-569b-4c15-8927-bf08bd194107-kube-api-access-q2fnc" pod "60ede993-569b-4c15-8927-bf08bd194107" (UID: "60ede993-569b-4c15-8927-bf08bd194107"). InnerVolumeSpecName "kube-api-access-q2fnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:39:23.579489 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:23.579464 5601 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60ede993-569b-4c15-8927-bf08bd194107-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:39:23.579489 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:23.579486 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2fnc\" (UniqueName: \"kubernetes.io/projected/60ede993-569b-4c15-8927-bf08bd194107-kube-api-access-q2fnc\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:39:24.248301 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:24.248262 5601 generic.go:356] "Generic (PLEG): container finished" podID="60ede993-569b-4c15-8927-bf08bd194107" containerID="069a86347fc66f1b0e09e4440083cbb7f6d322d499e84deb6a1c9d6d0f501808" exitCode=0 Nov 02 17:39:24.248818 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:24.248307 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr" event={"ID":"60ede993-569b-4c15-8927-bf08bd194107","Type":"ContainerDied","Data":"069a86347fc66f1b0e09e4440083cbb7f6d322d499e84deb6a1c9d6d0f501808"} Nov 02 17:39:24.248818 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:24.248335 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr" Nov 02 17:39:24.248818 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:24.248384 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr" event={"ID":"60ede993-569b-4c15-8927-bf08bd194107","Type":"ContainerDied","Data":"fcec384d633d755672d85a9d7de34209f9296cc14436c1ed80f9cfd89e0f50c3"} Nov 02 17:39:24.248818 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:24.248407 5601 scope.go:122] "RemoveContainer" containerID="069a86347fc66f1b0e09e4440083cbb7f6d322d499e84deb6a1c9d6d0f501808" Nov 02 17:39:24.257888 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:24.257871 5601 scope.go:122] "RemoveContainer" containerID="069a86347fc66f1b0e09e4440083cbb7f6d322d499e84deb6a1c9d6d0f501808" Nov 02 17:39:24.258194 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:39:24.258169 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"069a86347fc66f1b0e09e4440083cbb7f6d322d499e84deb6a1c9d6d0f501808\": container with ID starting with 069a86347fc66f1b0e09e4440083cbb7f6d322d499e84deb6a1c9d6d0f501808 not found: ID does not exist" containerID="069a86347fc66f1b0e09e4440083cbb7f6d322d499e84deb6a1c9d6d0f501808" Nov 02 17:39:24.258244 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:24.258206 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"069a86347fc66f1b0e09e4440083cbb7f6d322d499e84deb6a1c9d6d0f501808"} err="failed to get container status \"069a86347fc66f1b0e09e4440083cbb7f6d322d499e84deb6a1c9d6d0f501808\": rpc error: code = NotFound desc = could not find container \"069a86347fc66f1b0e09e4440083cbb7f6d322d499e84deb6a1c9d6d0f501808\": container with ID starting with 069a86347fc66f1b0e09e4440083cbb7f6d322d499e84deb6a1c9d6d0f501808 not found: ID does not exist" Nov 02 17:39:24.270206 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:24.270178 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr"] Nov 02 17:39:24.273963 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:24.273937 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lxmdr"] Nov 02 17:39:25.424036 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:39:25.424005 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ede993-569b-4c15-8927-bf08bd194107" path="/var/lib/kubelet/pods/60ede993-569b-4c15-8927-bf08bd194107/volumes" Nov 02 17:39:34.484244 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:39:34.484203 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:40:04.490405 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:40:04.490360 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:40:21.956380 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:21.956345 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4"] Nov 02 17:40:21.957054 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:21.956935 5601 cpu_manager.go:414] "containerMap: removing container" podUID="60ede993-569b-4c15-8927-bf08bd194107" containerName="manager" Nov 02 17:40:21.957054 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:21.956956 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ede993-569b-4c15-8927-bf08bd194107" containerName="manager" Nov 02 17:40:21.957174 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:21.957060 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ede993-569b-4c15-8927-bf08bd194107" containerName="manager" Nov 02 17:40:21.960450 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:21.960429 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:21.973855 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:21.973826 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4"] Nov 02 17:40:22.004002 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.003966 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c32fa6f3-9665-4b3b-b093-22435190829b-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.004162 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.004008 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c32fa6f3-9665-4b3b-b093-22435190829b-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.004162 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.004025 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c32fa6f3-9665-4b3b-b093-22435190829b-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.004239 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.004184 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c32fa6f3-9665-4b3b-b093-22435190829b-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.004239 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.004213 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5mhh\" (UniqueName: \"kubernetes.io/projected/c32fa6f3-9665-4b3b-b093-22435190829b-kube-api-access-m5mhh\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.004304 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.004241 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c32fa6f3-9665-4b3b-b093-22435190829b-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.004337 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.004312 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c32fa6f3-9665-4b3b-b093-22435190829b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.004370 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.004345 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c32fa6f3-9665-4b3b-b093-22435190829b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.004442 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.004422 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c32fa6f3-9665-4b3b-b093-22435190829b-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.105151 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.105111 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c32fa6f3-9665-4b3b-b093-22435190829b-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.105151 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.105156 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c32fa6f3-9665-4b3b-b093-22435190829b-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.105363 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.105276 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c32fa6f3-9665-4b3b-b093-22435190829b-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.105363 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.105310 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c32fa6f3-9665-4b3b-b093-22435190829b-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.105452 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.105379 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c32fa6f3-9665-4b3b-b093-22435190829b-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.105452 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.105406 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5mhh\" (UniqueName: \"kubernetes.io/projected/c32fa6f3-9665-4b3b-b093-22435190829b-kube-api-access-m5mhh\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.105452 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.105440 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c32fa6f3-9665-4b3b-b093-22435190829b-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.105710 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.105670 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c32fa6f3-9665-4b3b-b093-22435190829b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.105710 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.105700 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c32fa6f3-9665-4b3b-b093-22435190829b-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.105846 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.105751 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c32fa6f3-9665-4b3b-b093-22435190829b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.105846 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.105782 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c32fa6f3-9665-4b3b-b093-22435190829b-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.105961 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.105847 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c32fa6f3-9665-4b3b-b093-22435190829b-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.105961 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.105856 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c32fa6f3-9665-4b3b-b093-22435190829b-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.106290 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.106270 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c32fa6f3-9665-4b3b-b093-22435190829b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.107627 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.107587 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c32fa6f3-9665-4b3b-b093-22435190829b-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.108026 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.108005 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c32fa6f3-9665-4b3b-b093-22435190829b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.114161 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.114140 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5mhh\" (UniqueName: \"kubernetes.io/projected/c32fa6f3-9665-4b3b-b093-22435190829b-kube-api-access-m5mhh\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.114641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.114585 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c32fa6f3-9665-4b3b-b093-22435190829b-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-jv8f4\" (UID: \"c32fa6f3-9665-4b3b-b093-22435190829b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.277988 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.277907 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:22.555839 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.555665 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4"] Nov 02 17:40:22.559041 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:40:22.559008 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc32fa6f3_9665_4b3b_b093_22435190829b.slice/crio-b6b5da80e9a98ec8be6e564d919275117533a2910756153836e5f8e157c0dacc WatchSource:0}: Error finding container b6b5da80e9a98ec8be6e564d919275117533a2910756153836e5f8e157c0dacc: Status 404 returned error can't find the container with id b6b5da80e9a98ec8be6e564d919275117533a2910756153836e5f8e157c0dacc Nov 02 17:40:22.560954 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.560917 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:40:22.561062 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.560984 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:40:22.561062 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:22.561014 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:40:23.566471 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:23.566427 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" event={"ID":"c32fa6f3-9665-4b3b-b093-22435190829b","Type":"ContainerStarted","Data":"088eaf5b9295a790e5f28b6ea57ab9c79e2704a079c263c2ae931b29e807858f"} Nov 02 17:40:23.566958 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:23.566484 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" event={"ID":"c32fa6f3-9665-4b3b-b093-22435190829b","Type":"ContainerStarted","Data":"b6b5da80e9a98ec8be6e564d919275117533a2910756153836e5f8e157c0dacc"} Nov 02 17:40:23.591904 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:23.591852 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" podStartSLOduration=2.591838778 podStartE2EDuration="2.591838778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:23.590291296 +0000 UTC m=+388.281345309" watchObservedRunningTime="2026-11-02 17:40:23.591838778 +0000 UTC m=+388.282892789" Nov 02 17:40:23.965434 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:23.965373 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68d65c5784-6mkqh"] Nov 02 17:40:23.965924 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:23.965892 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-68d65c5784-6mkqh" podUID="821376c6-e692-4455-bfd0-d7d4158fa006" containerName="authorino" containerID="cri-o://e570bc049a995c176bf536f454b8aa8ad7db6c046a82450a2e307f563ff2d3cc" gracePeriod=30 Nov 02 17:40:24.265799 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:24.265770 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68d65c5784-6mkqh" Nov 02 17:40:24.278584 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:24.278557 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:24.280035 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:24.280015 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-jv8f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:40:24.280087 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:24.280062 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" podUID="c32fa6f3-9665-4b3b-b093-22435190829b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 17:40:24.328445 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:24.328413 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821376c6-e692-4455-bfd0-d7d4158fa006-kube-api-access-rrz5l\" (UniqueName: \"kubernetes.io/projected/821376c6-e692-4455-bfd0-d7d4158fa006-kube-api-access-rrz5l\") pod \"821376c6-e692-4455-bfd0-d7d4158fa006\" (UID: \"821376c6-e692-4455-bfd0-d7d4158fa006\") " Nov 02 17:40:24.328599 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:24.328512 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/821376c6-e692-4455-bfd0-d7d4158fa006-tls-cert\" (UniqueName: \"kubernetes.io/secret/821376c6-e692-4455-bfd0-d7d4158fa006-tls-cert\") pod \"821376c6-e692-4455-bfd0-d7d4158fa006\" (UID: \"821376c6-e692-4455-bfd0-d7d4158fa006\") " Nov 02 17:40:24.330489 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:24.330459 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821376c6-e692-4455-bfd0-d7d4158fa006-kube-api-access-rrz5l" pod "821376c6-e692-4455-bfd0-d7d4158fa006" (UID: "821376c6-e692-4455-bfd0-d7d4158fa006"). InnerVolumeSpecName "kube-api-access-rrz5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:40:24.363186 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:24.363152 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/821376c6-e692-4455-bfd0-d7d4158fa006-tls-cert" pod "821376c6-e692-4455-bfd0-d7d4158fa006" (UID: "821376c6-e692-4455-bfd0-d7d4158fa006"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:40:24.430025 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:24.429991 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrz5l\" (UniqueName: \"kubernetes.io/projected/821376c6-e692-4455-bfd0-d7d4158fa006-kube-api-access-rrz5l\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:40:24.430025 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:24.430022 5601 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/821376c6-e692-4455-bfd0-d7d4158fa006-tls-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:40:24.572181 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:24.572085 5601 generic.go:356] "Generic (PLEG): container finished" podID="821376c6-e692-4455-bfd0-d7d4158fa006" containerID="e570bc049a995c176bf536f454b8aa8ad7db6c046a82450a2e307f563ff2d3cc" exitCode=0 Nov 02 17:40:24.572181 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:24.572147 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68d65c5784-6mkqh" Nov 02 17:40:24.572181 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:24.572168 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68d65c5784-6mkqh" event={"ID":"821376c6-e692-4455-bfd0-d7d4158fa006","Type":"ContainerDied","Data":"e570bc049a995c176bf536f454b8aa8ad7db6c046a82450a2e307f563ff2d3cc"} Nov 02 17:40:24.572752 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:24.572220 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68d65c5784-6mkqh" event={"ID":"821376c6-e692-4455-bfd0-d7d4158fa006","Type":"ContainerDied","Data":"89c969cd796365f7f920d50984379c9ba6b2ccdeaa553fff1f471c95d1f1e15e"} Nov 02 17:40:24.572752 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:24.572251 5601 scope.go:122] "RemoveContainer" containerID="e570bc049a995c176bf536f454b8aa8ad7db6c046a82450a2e307f563ff2d3cc" Nov 02 17:40:24.581829 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:24.581810 5601 scope.go:122] "RemoveContainer" containerID="e570bc049a995c176bf536f454b8aa8ad7db6c046a82450a2e307f563ff2d3cc" Nov 02 17:40:24.582073 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:40:24.582056 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e570bc049a995c176bf536f454b8aa8ad7db6c046a82450a2e307f563ff2d3cc\": container with ID starting with e570bc049a995c176bf536f454b8aa8ad7db6c046a82450a2e307f563ff2d3cc not found: ID does not exist" containerID="e570bc049a995c176bf536f454b8aa8ad7db6c046a82450a2e307f563ff2d3cc" Nov 02 17:40:24.582113 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:24.582080 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e570bc049a995c176bf536f454b8aa8ad7db6c046a82450a2e307f563ff2d3cc"} err="failed to get container status \"e570bc049a995c176bf536f454b8aa8ad7db6c046a82450a2e307f563ff2d3cc\": rpc error: code = NotFound desc = could not find container \"e570bc049a995c176bf536f454b8aa8ad7db6c046a82450a2e307f563ff2d3cc\": container with ID starting with e570bc049a995c176bf536f454b8aa8ad7db6c046a82450a2e307f563ff2d3cc not found: ID does not exist" Nov 02 17:40:24.594527 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:24.594501 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68d65c5784-6mkqh"] Nov 02 17:40:24.599417 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:24.599394 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-68d65c5784-6mkqh"] Nov 02 17:40:25.279121 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:25.279085 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-jv8f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:40:25.279304 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:25.279151 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" podUID="c32fa6f3-9665-4b3b-b093-22435190829b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 17:40:25.425188 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:25.425155 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="821376c6-e692-4455-bfd0-d7d4158fa006" path="/var/lib/kubelet/pods/821376c6-e692-4455-bfd0-d7d4158fa006/volumes" Nov 02 17:40:26.278544 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:26.278511 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-jv8f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:40:26.279058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:26.278564 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" podUID="c32fa6f3-9665-4b3b-b093-22435190829b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 17:40:27.278422 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:27.278389 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-jv8f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:40:27.278592 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:27.278438 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" podUID="c32fa6f3-9665-4b3b-b093-22435190829b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 17:40:28.279099 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:28.279065 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-jv8f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:40:28.279493 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:28.279118 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" podUID="c32fa6f3-9665-4b3b-b093-22435190829b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 17:40:29.278670 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:29.278631 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-jv8f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:40:29.278871 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:29.278692 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" podUID="c32fa6f3-9665-4b3b-b093-22435190829b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 17:40:30.746638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:30.746595 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:30.747107 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:30.747029 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:30.747820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:40:30.747798 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" Nov 02 17:40:34.483279 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:40:34.483241 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:41:02.244604 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.244566 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv"] Nov 02 17:41:02.245326 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.245297 5601 cpu_manager.go:414] "containerMap: removing container" podUID="821376c6-e692-4455-bfd0-d7d4158fa006" containerName="authorino" Nov 02 17:41:02.245326 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.245319 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="821376c6-e692-4455-bfd0-d7d4158fa006" containerName="authorino" Nov 02 17:41:02.245771 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.245482 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="821376c6-e692-4455-bfd0-d7d4158fa006" containerName="authorino" Nov 02 17:41:02.250215 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.250196 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" Nov 02 17:41:02.259085 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.259060 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-38f4eea3-serving-cert\"" Nov 02 17:41:02.259199 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.259094 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tqgzn\"" Nov 02 17:41:02.259199 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.259094 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 17:41:02.264904 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.264881 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv"] Nov 02 17:41:02.376403 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.376363 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-metrics-certs\") pod \"maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv\" (UID: \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" Nov 02 17:41:02.376634 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.376501 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-maas-api-tls\") pod \"maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv\" (UID: \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" Nov 02 17:41:02.376634 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.376547 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-openshift-service-ca\") pod \"maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv\" (UID: \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" Nov 02 17:41:02.376634 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.376577 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxkd2\" (UniqueName: \"kubernetes.io/projected/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-kube-api-access-pxkd2\") pod \"maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv\" (UID: \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" Nov 02 17:41:02.477720 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.477682 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-metrics-certs\") pod \"maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv\" (UID: \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" Nov 02 17:41:02.477925 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.477740 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-maas-api-tls\") pod \"maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv\" (UID: \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" Nov 02 17:41:02.477925 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.477767 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-openshift-service-ca\") pod \"maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv\" (UID: \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" Nov 02 17:41:02.477925 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.477786 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxkd2\" (UniqueName: \"kubernetes.io/projected/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-kube-api-access-pxkd2\") pod \"maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv\" (UID: \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" Nov 02 17:41:02.478641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.478585 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-openshift-service-ca\") pod \"maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv\" (UID: \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" Nov 02 17:41:02.480406 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.480378 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-maas-api-tls\") pod \"maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv\" (UID: \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" Nov 02 17:41:02.480501 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.480439 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-metrics-certs\") pod \"maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv\" (UID: \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" Nov 02 17:41:02.485894 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.485869 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxkd2\" (UniqueName: \"kubernetes.io/projected/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-kube-api-access-pxkd2\") pod \"maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv\" (UID: \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" Nov 02 17:41:02.543734 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.543652 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb"] Nov 02 17:41:02.547367 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.547343 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb" Nov 02 17:41:02.555939 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.555817 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-38f4eea3\"" Nov 02 17:41:02.555939 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.555851 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-38f4eea3-dockercfg-h8bql\"" Nov 02 17:41:02.557975 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.557948 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb"] Nov 02 17:41:02.563386 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.563362 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" Nov 02 17:41:02.579647 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.579518 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/378ef976-7343-4b07-8d8e-9db56e00ae94-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb\" (UID: \"378ef976-7343-4b07-8d8e-9db56e00ae94\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb" Nov 02 17:41:02.579759 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.579675 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7xtq\" (UniqueName: \"kubernetes.io/projected/378ef976-7343-4b07-8d8e-9db56e00ae94-kube-api-access-q7xtq\") pod \"payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb\" (UID: \"378ef976-7343-4b07-8d8e-9db56e00ae94\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb" Nov 02 17:41:02.596557 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.596517 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq"] Nov 02 17:41:02.600475 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.600446 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq" Nov 02 17:41:02.611979 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.611907 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq"] Nov 02 17:41:02.680743 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.680707 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ad6740-6d33-412d-8af3-ba486b5848b9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq\" (UID: \"67ad6740-6d33-412d-8af3-ba486b5848b9\") " pod="openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq" Nov 02 17:41:02.680910 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.680777 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/378ef976-7343-4b07-8d8e-9db56e00ae94-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb\" (UID: \"378ef976-7343-4b07-8d8e-9db56e00ae94\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb" Nov 02 17:41:02.680996 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.680906 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bblx\" (UniqueName: \"kubernetes.io/projected/67ad6740-6d33-412d-8af3-ba486b5848b9-kube-api-access-6bblx\") pod \"payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq\" (UID: \"67ad6740-6d33-412d-8af3-ba486b5848b9\") " pod="openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq" Nov 02 17:41:02.681053 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.681002 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7xtq\" (UniqueName: \"kubernetes.io/projected/378ef976-7343-4b07-8d8e-9db56e00ae94-kube-api-access-q7xtq\") pod \"payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb\" (UID: \"378ef976-7343-4b07-8d8e-9db56e00ae94\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb" Nov 02 17:41:02.681570 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.681543 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/378ef976-7343-4b07-8d8e-9db56e00ae94-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb\" (UID: \"378ef976-7343-4b07-8d8e-9db56e00ae94\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb" Nov 02 17:41:02.689690 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.689663 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7xtq\" (UniqueName: \"kubernetes.io/projected/378ef976-7343-4b07-8d8e-9db56e00ae94-kube-api-access-q7xtq\") pod \"payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb\" (UID: \"378ef976-7343-4b07-8d8e-9db56e00ae94\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb" Nov 02 17:41:02.782002 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.781799 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bblx\" (UniqueName: \"kubernetes.io/projected/67ad6740-6d33-412d-8af3-ba486b5848b9-kube-api-access-6bblx\") pod \"payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq\" (UID: \"67ad6740-6d33-412d-8af3-ba486b5848b9\") " pod="openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq" Nov 02 17:41:02.782002 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.781882 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ad6740-6d33-412d-8af3-ba486b5848b9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq\" (UID: \"67ad6740-6d33-412d-8af3-ba486b5848b9\") " pod="openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq" Nov 02 17:41:02.782692 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.782671 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ad6740-6d33-412d-8af3-ba486b5848b9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq\" (UID: \"67ad6740-6d33-412d-8af3-ba486b5848b9\") " pod="openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq" Nov 02 17:41:02.791794 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.791761 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bblx\" (UniqueName: \"kubernetes.io/projected/67ad6740-6d33-412d-8af3-ba486b5848b9-kube-api-access-6bblx\") pod \"payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq\" (UID: \"67ad6740-6d33-412d-8af3-ba486b5848b9\") " pod="openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq" Nov 02 17:41:02.850671 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.850641 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv"] Nov 02 17:41:02.853160 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:41:02.853134 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5feaad83_55b3_4503_8eb7_9f5c33f38f9e.slice/crio-d23eb7ba9b2f1a43cff54cfea5aca2afeb5361da7e8d09b94a8b10e61326ed3b WatchSource:0}: Error finding container d23eb7ba9b2f1a43cff54cfea5aca2afeb5361da7e8d09b94a8b10e61326ed3b: Status 404 returned error can't find the container with id d23eb7ba9b2f1a43cff54cfea5aca2afeb5361da7e8d09b94a8b10e61326ed3b Nov 02 17:41:02.854433 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.854416 5601 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:41:02.863080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.863012 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb" Nov 02 17:41:02.916913 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:02.916876 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq" Nov 02 17:41:03.148771 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:03.148746 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb"] Nov 02 17:41:03.151790 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:41:03.151741 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378ef976_7343_4b07_8d8e_9db56e00ae94.slice/crio-897bdd1e5941484bc052e7bb3d1926580ae55865fb785a3f3119ce1eaf056c16 WatchSource:0}: Error finding container 897bdd1e5941484bc052e7bb3d1926580ae55865fb785a3f3119ce1eaf056c16: Status 404 returned error can't find the container with id 897bdd1e5941484bc052e7bb3d1926580ae55865fb785a3f3119ce1eaf056c16 Nov 02 17:41:03.218241 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:03.218201 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq"] Nov 02 17:41:03.768999 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:03.768958 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" event={"ID":"5feaad83-55b3-4503-8eb7-9f5c33f38f9e","Type":"ContainerStarted","Data":"9040670e27e394cffeddd06571ea4c78739469d58580bb2a61ff92dfb2aa392f"} Nov 02 17:41:03.769513 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:03.769029 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" event={"ID":"5feaad83-55b3-4503-8eb7-9f5c33f38f9e","Type":"ContainerStarted","Data":"d23eb7ba9b2f1a43cff54cfea5aca2afeb5361da7e8d09b94a8b10e61326ed3b"} Nov 02 17:41:03.769513 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:03.769166 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" Nov 02 17:41:03.771585 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:03.771502 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb" event={"ID":"378ef976-7343-4b07-8d8e-9db56e00ae94","Type":"ContainerStarted","Data":"897bdd1e5941484bc052e7bb3d1926580ae55865fb785a3f3119ce1eaf056c16"} Nov 02 17:41:03.774007 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:03.773976 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq" event={"ID":"67ad6740-6d33-412d-8af3-ba486b5848b9","Type":"ContainerStarted","Data":"1d70a9ad475763e9f1b2ced548eaf82b0fde52cb84e1e51adb6a7bdb75635da3"} Nov 02 17:41:03.795299 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:03.793983 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" podStartSLOduration=1.323415704 podStartE2EDuration="1.793921523s" totalImagesPullingTime="470.505819ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:03.792141603 +0000 UTC m=+428.483195617" watchObservedRunningTime="2026-11-02 17:41:03.793921523 +0000 UTC m=+428.484975535" Nov 02 17:41:03.941097 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:03.941052 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm"] Nov 02 17:41:03.945084 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:03.945061 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm" Nov 02 17:41:03.957987 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:03.957958 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ad831efe\"" Nov 02 17:41:03.962141 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:03.962112 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm"] Nov 02 17:41:03.994253 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:03.994210 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mv5r\" (UniqueName: \"kubernetes.io/projected/ea9816f4-bad9-4622-a16c-6f7846f72c9a-kube-api-access-9mv5r\") pod \"payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm\" (UID: \"ea9816f4-bad9-4622-a16c-6f7846f72c9a\") " pod="openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm" Nov 02 17:41:03.994417 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:03.994380 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea9816f4-bad9-4622-a16c-6f7846f72c9a-plugins-config-volume\") pod \"payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm\" (UID: \"ea9816f4-bad9-4622-a16c-6f7846f72c9a\") " pod="openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm" Nov 02 17:41:04.095964 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:04.095868 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea9816f4-bad9-4622-a16c-6f7846f72c9a-plugins-config-volume\") pod \"payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm\" (UID: \"ea9816f4-bad9-4622-a16c-6f7846f72c9a\") " pod="openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm" Nov 02 17:41:04.095964 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:04.095964 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mv5r\" (UniqueName: \"kubernetes.io/projected/ea9816f4-bad9-4622-a16c-6f7846f72c9a-kube-api-access-9mv5r\") pod \"payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm\" (UID: \"ea9816f4-bad9-4622-a16c-6f7846f72c9a\") " pod="openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm" Nov 02 17:41:04.097041 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:04.096982 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea9816f4-bad9-4622-a16c-6f7846f72c9a-plugins-config-volume\") pod \"payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm\" (UID: \"ea9816f4-bad9-4622-a16c-6f7846f72c9a\") " pod="openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm" Nov 02 17:41:04.123002 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:04.122936 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mv5r\" (UniqueName: \"kubernetes.io/projected/ea9816f4-bad9-4622-a16c-6f7846f72c9a-kube-api-access-9mv5r\") pod \"payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm\" (UID: \"ea9816f4-bad9-4622-a16c-6f7846f72c9a\") " pod="openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm" Nov 02 17:41:04.266594 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:04.266549 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm" Nov 02 17:41:04.494191 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:41:04.494148 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:41:04.586231 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:04.586143 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm"] Nov 02 17:41:04.588006 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:41:04.587973 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9816f4_bad9_4622_a16c_6f7846f72c9a.slice/crio-e9b17e1206c1833d87ef44f094bf6ad6e908d7cc6f2b6467ab04a26e42e74bbe WatchSource:0}: Error finding container e9b17e1206c1833d87ef44f094bf6ad6e908d7cc6f2b6467ab04a26e42e74bbe: Status 404 returned error can't find the container with id e9b17e1206c1833d87ef44f094bf6ad6e908d7cc6f2b6467ab04a26e42e74bbe Nov 02 17:41:04.783787 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:04.783717 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm" event={"ID":"ea9816f4-bad9-4622-a16c-6f7846f72c9a","Type":"ContainerStarted","Data":"e9b17e1206c1833d87ef44f094bf6ad6e908d7cc6f2b6467ab04a26e42e74bbe"} Nov 02 17:41:05.035331 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.035242 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv"] Nov 02 17:41:05.052472 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.052428 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv"] Nov 02 17:41:05.052714 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.052513 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv" Nov 02 17:41:05.108168 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.108128 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04c7b10-dac8-4ba7-a86b-801eec1f95fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv\" (UID: \"c04c7b10-dac8-4ba7-a86b-801eec1f95fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv" Nov 02 17:41:05.108377 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.108185 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmm4r\" (UniqueName: \"kubernetes.io/projected/c04c7b10-dac8-4ba7-a86b-801eec1f95fd-kube-api-access-fmm4r\") pod \"payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv\" (UID: \"c04c7b10-dac8-4ba7-a86b-801eec1f95fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv" Nov 02 17:41:05.124503 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.124463 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0ae9c41e\"" Nov 02 17:41:05.127855 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.125216 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0ae9c41e-dockercfg-rmksk\"" Nov 02 17:41:05.211144 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.209842 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04c7b10-dac8-4ba7-a86b-801eec1f95fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv\" (UID: \"c04c7b10-dac8-4ba7-a86b-801eec1f95fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv" Nov 02 17:41:05.211144 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.209909 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmm4r\" (UniqueName: \"kubernetes.io/projected/c04c7b10-dac8-4ba7-a86b-801eec1f95fd-kube-api-access-fmm4r\") pod \"payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv\" (UID: \"c04c7b10-dac8-4ba7-a86b-801eec1f95fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv" Nov 02 17:41:05.211144 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.211092 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04c7b10-dac8-4ba7-a86b-801eec1f95fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv\" (UID: \"c04c7b10-dac8-4ba7-a86b-801eec1f95fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv" Nov 02 17:41:05.223575 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.223500 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmm4r\" (UniqueName: \"kubernetes.io/projected/c04c7b10-dac8-4ba7-a86b-801eec1f95fd-kube-api-access-fmm4r\") pod \"payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv\" (UID: \"c04c7b10-dac8-4ba7-a86b-801eec1f95fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv" Nov 02 17:41:05.372737 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.372655 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv" Nov 02 17:41:05.513123 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.513085 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf"] Nov 02 17:41:05.520403 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.520375 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" Nov 02 17:41:05.523908 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.523841 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf"] Nov 02 17:41:05.526358 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.526336 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0ae9c41e-serving-cert\"" Nov 02 17:41:05.614789 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.614389 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcdl6\" (UniqueName: \"kubernetes.io/projected/21d45c50-5921-4206-8fec-1c07a1dd3bef-kube-api-access-lcdl6\") pod \"maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf\" (UID: \"21d45c50-5921-4206-8fec-1c07a1dd3bef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" Nov 02 17:41:05.614789 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.614456 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21d45c50-5921-4206-8fec-1c07a1dd3bef-openshift-service-ca\") pod \"maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf\" (UID: \"21d45c50-5921-4206-8fec-1c07a1dd3bef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" Nov 02 17:41:05.614789 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.614509 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21d45c50-5921-4206-8fec-1c07a1dd3bef-maas-api-tls\") pod \"maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf\" (UID: \"21d45c50-5921-4206-8fec-1c07a1dd3bef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" Nov 02 17:41:05.614789 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.614559 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21d45c50-5921-4206-8fec-1c07a1dd3bef-metrics-certs\") pod \"maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf\" (UID: \"21d45c50-5921-4206-8fec-1c07a1dd3bef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" Nov 02 17:41:05.661089 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.661007 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv"] Nov 02 17:41:05.671893 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:41:05.671854 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04c7b10_dac8_4ba7_a86b_801eec1f95fd.slice/crio-0b4e061daecf36eb48777500c4d1b560c6d6664edf8c4c5207c04721779d6a32 WatchSource:0}: Error finding container 0b4e061daecf36eb48777500c4d1b560c6d6664edf8c4c5207c04721779d6a32: Status 404 returned error can't find the container with id 0b4e061daecf36eb48777500c4d1b560c6d6664edf8c4c5207c04721779d6a32 Nov 02 17:41:05.715427 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.715391 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcdl6\" (UniqueName: \"kubernetes.io/projected/21d45c50-5921-4206-8fec-1c07a1dd3bef-kube-api-access-lcdl6\") pod \"maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf\" (UID: \"21d45c50-5921-4206-8fec-1c07a1dd3bef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" Nov 02 17:41:05.715627 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.715446 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21d45c50-5921-4206-8fec-1c07a1dd3bef-openshift-service-ca\") pod \"maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf\" (UID: \"21d45c50-5921-4206-8fec-1c07a1dd3bef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" Nov 02 17:41:05.715627 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.715545 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21d45c50-5921-4206-8fec-1c07a1dd3bef-maas-api-tls\") pod \"maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf\" (UID: \"21d45c50-5921-4206-8fec-1c07a1dd3bef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" Nov 02 17:41:05.715773 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.715734 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21d45c50-5921-4206-8fec-1c07a1dd3bef-metrics-certs\") pod \"maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf\" (UID: \"21d45c50-5921-4206-8fec-1c07a1dd3bef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" Nov 02 17:41:05.716576 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.716545 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21d45c50-5921-4206-8fec-1c07a1dd3bef-openshift-service-ca\") pod \"maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf\" (UID: \"21d45c50-5921-4206-8fec-1c07a1dd3bef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" Nov 02 17:41:05.726301 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.726269 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcdl6\" (UniqueName: \"kubernetes.io/projected/21d45c50-5921-4206-8fec-1c07a1dd3bef-kube-api-access-lcdl6\") pod \"maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf\" (UID: \"21d45c50-5921-4206-8fec-1c07a1dd3bef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" Nov 02 17:41:05.729890 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.729836 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21d45c50-5921-4206-8fec-1c07a1dd3bef-metrics-certs\") pod \"maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf\" (UID: \"21d45c50-5921-4206-8fec-1c07a1dd3bef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" Nov 02 17:41:05.731686 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.731655 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21d45c50-5921-4206-8fec-1c07a1dd3bef-maas-api-tls\") pod \"maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf\" (UID: \"21d45c50-5921-4206-8fec-1c07a1dd3bef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" Nov 02 17:41:05.790987 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.790941 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv" event={"ID":"c04c7b10-dac8-4ba7-a86b-801eec1f95fd","Type":"ContainerStarted","Data":"0b4e061daecf36eb48777500c4d1b560c6d6664edf8c4c5207c04721779d6a32"} Nov 02 17:41:05.843523 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:05.843484 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" Nov 02 17:41:06.137131 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.137099 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf"] Nov 02 17:41:06.141011 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:41:06.140983 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d45c50_5921_4206_8fec_1c07a1dd3bef.slice/crio-431c141d41e673e944d68d07fc494ac842c7f4c2d7bdd4ae9b9e216e15f354f3 WatchSource:0}: Error finding container 431c141d41e673e944d68d07fc494ac842c7f4c2d7bdd4ae9b9e216e15f354f3: Status 404 returned error can't find the container with id 431c141d41e673e944d68d07fc494ac842c7f4c2d7bdd4ae9b9e216e15f354f3 Nov 02 17:41:06.405164 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.405129 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d"] Nov 02 17:41:06.409302 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.409270 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" Nov 02 17:41:06.417168 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.416908 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-868a2ee9-serving-cert\"" Nov 02 17:41:06.420075 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.419610 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d"] Nov 02 17:41:06.524864 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.524830 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-openshift-service-ca\") pod \"maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d\" (UID: \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" Nov 02 17:41:06.524864 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.524864 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-metrics-certs\") pod \"maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d\" (UID: \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" Nov 02 17:41:06.525076 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.524990 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wwqc\" (UniqueName: \"kubernetes.io/projected/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-kube-api-access-7wwqc\") pod \"maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d\" (UID: \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" Nov 02 17:41:06.525076 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.525058 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-maas-api-tls\") pod \"maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d\" (UID: \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" Nov 02 17:41:06.626688 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.626647 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-openshift-service-ca\") pod \"maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d\" (UID: \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" Nov 02 17:41:06.626885 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.626694 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-metrics-certs\") pod \"maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d\" (UID: \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" Nov 02 17:41:06.626885 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.626736 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wwqc\" (UniqueName: \"kubernetes.io/projected/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-kube-api-access-7wwqc\") pod \"maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d\" (UID: \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" Nov 02 17:41:06.626885 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.626814 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-maas-api-tls\") pod \"maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d\" (UID: \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" Nov 02 17:41:06.627052 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:41:06.626966 5601 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-serving-cert: secret "maas-api-e2e-ait-868a2ee9-serving-cert" not found Nov 02 17:41:06.627101 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:41:06.627053 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-maas-api-tls podName:5366ed79-2236-4896-8ed6-b34b5d6f4cbc nodeName:}" failed. No retries permitted until 2026-11-02 17:41:07.127028441 +0000 UTC m=+431.818082452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-maas-api-tls") pod "maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" (UID: "5366ed79-2236-4896-8ed6-b34b5d6f4cbc") : secret "maas-api-e2e-ait-868a2ee9-serving-cert" not found Nov 02 17:41:06.627408 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.627382 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-openshift-service-ca\") pod \"maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d\" (UID: \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" Nov 02 17:41:06.629301 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.629281 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-metrics-certs\") pod \"maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d\" (UID: \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" Nov 02 17:41:06.640031 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.639997 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wwqc\" (UniqueName: \"kubernetes.io/projected/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-kube-api-access-7wwqc\") pod \"maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d\" (UID: \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" Nov 02 17:41:06.799253 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.799214 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm" event={"ID":"ea9816f4-bad9-4622-a16c-6f7846f72c9a","Type":"ContainerStarted","Data":"f5132738363890611778c6e359e4b6472cbea25197dc1574a2b87f6d7b7962e6"} Nov 02 17:41:06.799785 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.799345 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm" Nov 02 17:41:06.801100 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.801066 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" event={"ID":"21d45c50-5921-4206-8fec-1c07a1dd3bef","Type":"ContainerStarted","Data":"6c6a08eec52e4b33faafba74f7e1397c0457ecbd1ff0f969d11be9f088085bda"} Nov 02 17:41:06.801237 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.801129 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" Nov 02 17:41:06.801237 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.801154 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" event={"ID":"21d45c50-5921-4206-8fec-1c07a1dd3bef","Type":"ContainerStarted","Data":"431c141d41e673e944d68d07fc494ac842c7f4c2d7bdd4ae9b9e216e15f354f3"} Nov 02 17:41:06.802769 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.802739 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb" event={"ID":"378ef976-7343-4b07-8d8e-9db56e00ae94","Type":"ContainerStarted","Data":"a8f7c744ee33d1912e20f3afafa2b2fc96e31c218a728cc332d72e512a267b87"} Nov 02 17:41:06.802882 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.802813 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb" Nov 02 17:41:06.804688 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.804572 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv" event={"ID":"c04c7b10-dac8-4ba7-a86b-801eec1f95fd","Type":"ContainerStarted","Data":"7fcaa9187dd24bc29a12e01297089dc0cf027a5ae663f06a9bb3adcf4bb79b57"} Nov 02 17:41:06.804688 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.804654 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv" Nov 02 17:41:06.806780 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.806753 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq" event={"ID":"67ad6740-6d33-412d-8af3-ba486b5848b9","Type":"ContainerStarted","Data":"0b98f31af2d89484b7ca859eac3eab50dac2ac971baabf09722691476d60e6c5"} Nov 02 17:41:06.806908 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.806828 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq" Nov 02 17:41:06.818325 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.818272 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm" podStartSLOduration=2.256872238 podStartE2EDuration="3.818258713s" totalImagesPullingTime="1.561386475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:06.815447023 +0000 UTC m=+431.506501037" watchObservedRunningTime="2026-11-02 17:41:06.818258713 +0000 UTC m=+431.509312719" Nov 02 17:41:06.834462 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.834390 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq" podStartSLOduration=2.049132574 podStartE2EDuration="4.834370869s" totalImagesPullingTime="2.785238295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:06.833526423 +0000 UTC m=+431.524580439" watchObservedRunningTime="2026-11-02 17:41:06.834370869 +0000 UTC m=+431.525424882" Nov 02 17:41:06.850571 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.850510 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb" podStartSLOduration=2.000656326 podStartE2EDuration="4.850494112s" totalImagesPullingTime="2.849837786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:06.849420126 +0000 UTC m=+431.540474151" watchObservedRunningTime="2026-11-02 17:41:06.850494112 +0000 UTC m=+431.541548158" Nov 02 17:41:06.873416 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.873362 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv" podStartSLOduration=1.397726523 podStartE2EDuration="1.873346337s" totalImagesPullingTime="475.619814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:06.872393665 +0000 UTC m=+431.563447689" watchObservedRunningTime="2026-11-02 17:41:06.873346337 +0000 UTC m=+431.564400348" Nov 02 17:41:06.890863 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:06.890806 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" podStartSLOduration=1.496109563 podStartE2EDuration="1.890785621s" totalImagesPullingTime="394.676058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:06.889417087 +0000 UTC m=+431.580471098" watchObservedRunningTime="2026-11-02 17:41:06.890785621 +0000 UTC m=+431.581839636" Nov 02 17:41:07.131149 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:07.131058 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-maas-api-tls\") pod \"maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d\" (UID: \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" Nov 02 17:41:07.133465 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:07.133435 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-maas-api-tls\") pod \"maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d\" (UID: \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" Nov 02 17:41:07.326865 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:07.326832 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" Nov 02 17:41:07.609069 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:07.609036 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d"] Nov 02 17:41:07.610968 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:41:07.610940 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5366ed79_2236_4896_8ed6_b34b5d6f4cbc.slice/crio-39a2be6b037b367a581bdb4eaf8ce09bdd0303bde090ec5cb1b93da3b141e196 WatchSource:0}: Error finding container 39a2be6b037b367a581bdb4eaf8ce09bdd0303bde090ec5cb1b93da3b141e196: Status 404 returned error can't find the container with id 39a2be6b037b367a581bdb4eaf8ce09bdd0303bde090ec5cb1b93da3b141e196 Nov 02 17:41:07.813408 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:07.813366 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" event={"ID":"5366ed79-2236-4896-8ed6-b34b5d6f4cbc","Type":"ContainerStarted","Data":"39a2be6b037b367a581bdb4eaf8ce09bdd0303bde090ec5cb1b93da3b141e196"} Nov 02 17:41:08.820744 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:08.820709 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" event={"ID":"5366ed79-2236-4896-8ed6-b34b5d6f4cbc","Type":"ContainerStarted","Data":"89984fc515b4a00711ad9b6e14a88d3f57a584c8306699e2308e381446d5bc37"} Nov 02 17:41:08.821195 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:08.820866 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" Nov 02 17:41:08.839577 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:08.839523 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" podStartSLOduration=2.426956718 podStartE2EDuration="2.839509825s" totalImagesPullingTime="412.553107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:08.838041679 +0000 UTC m=+433.529095687" watchObservedRunningTime="2026-11-02 17:41:08.839509825 +0000 UTC m=+433.530563837" Nov 02 17:41:09.789594 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:09.789566 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" Nov 02 17:41:12.819718 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:12.819686 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" Nov 02 17:41:13.293899 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:13.293860 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd"] Nov 02 17:41:13.306603 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:13.306568 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" Nov 02 17:41:13.312399 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:13.312371 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd"] Nov 02 17:41:13.312649 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:13.312474 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-55f9886f-serving-cert\"" Nov 02 17:41:13.405191 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:13.405154 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k227m\" (UniqueName: \"kubernetes.io/projected/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-kube-api-access-k227m\") pod \"maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd\" (UID: \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" Nov 02 17:41:13.405368 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:13.405205 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-metrics-certs\") pod \"maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd\" (UID: \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" Nov 02 17:41:13.405433 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:13.405382 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-maas-api-tls\") pod \"maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd\" (UID: \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" Nov 02 17:41:13.405479 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:13.405440 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-openshift-service-ca\") pod \"maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd\" (UID: \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" Nov 02 17:41:13.506431 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:13.506393 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-openshift-service-ca\") pod \"maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd\" (UID: \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" Nov 02 17:41:13.506645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:13.506448 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k227m\" (UniqueName: \"kubernetes.io/projected/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-kube-api-access-k227m\") pod \"maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd\" (UID: \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" Nov 02 17:41:13.506645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:13.506489 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-metrics-certs\") pod \"maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd\" (UID: \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" Nov 02 17:41:13.506645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:13.506550 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-maas-api-tls\") pod \"maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd\" (UID: \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" Nov 02 17:41:13.507184 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:13.507154 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-openshift-service-ca\") pod \"maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd\" (UID: \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" Nov 02 17:41:13.509181 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:13.509155 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-maas-api-tls\") pod \"maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd\" (UID: \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" Nov 02 17:41:13.509181 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:13.509168 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-metrics-certs\") pod \"maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd\" (UID: \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" Nov 02 17:41:13.517127 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:13.517104 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k227m\" (UniqueName: \"kubernetes.io/projected/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-kube-api-access-k227m\") pod \"maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd\" (UID: \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" Nov 02 17:41:13.628797 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:13.628730 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" Nov 02 17:41:13.926646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:13.926604 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd"] Nov 02 17:41:13.929014 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:41:13.928988 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2cd242_0c2f_44bb_85e6_cf5a3a6d2782.slice/crio-724bc2b81e92b907f087144382865bda5346f6350d08e34b07d992719e9ffb38 WatchSource:0}: Error finding container 724bc2b81e92b907f087144382865bda5346f6350d08e34b07d992719e9ffb38: Status 404 returned error can't find the container with id 724bc2b81e92b907f087144382865bda5346f6350d08e34b07d992719e9ffb38 Nov 02 17:41:14.833303 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:14.833275 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" Nov 02 17:41:14.857263 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:14.857222 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" event={"ID":"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782","Type":"ContainerStarted","Data":"0a631347c644bc1ed69b88cbdcb24629f5699885501f674b6ecd29dc714775f3"} Nov 02 17:41:14.857263 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:14.857278 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" event={"ID":"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782","Type":"ContainerStarted","Data":"724bc2b81e92b907f087144382865bda5346f6350d08e34b07d992719e9ffb38"} Nov 02 17:41:14.857565 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:14.857366 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" Nov 02 17:41:17.817176 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:17.817134 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq" Nov 02 17:41:17.817595 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:17.817320 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb" Nov 02 17:41:17.817595 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:17.817360 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv" Nov 02 17:41:17.817595 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:17.817483 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm" Nov 02 17:41:17.836237 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:17.836178 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" podStartSLOduration=4.417901356 podStartE2EDuration="4.836162457s" totalImagesPullingTime="418.261101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:14.880786465 +0000 UTC m=+439.571840476" watchObservedRunningTime="2026-11-02 17:41:17.836162457 +0000 UTC m=+442.527216472" Nov 02 17:41:20.867757 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:20.867727 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" Nov 02 17:41:27.026492 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:27.026452 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq"] Nov 02 17:41:27.034907 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:27.034872 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb"] Nov 02 17:41:28.774666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:28.774629 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv"] Nov 02 17:41:28.775451 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:28.775418 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" podUID="5feaad83-55b3-4503-8eb7-9f5c33f38f9e" containerName="maas-api" containerID="cri-o://9040670e27e394cffeddd06571ea4c78739469d58580bb2a61ff92dfb2aa392f" gracePeriod=30 Nov 02 17:41:28.955029 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:28.954996 5601 generic.go:356] "Generic (PLEG): container finished" podID="5feaad83-55b3-4503-8eb7-9f5c33f38f9e" containerID="9040670e27e394cffeddd06571ea4c78739469d58580bb2a61ff92dfb2aa392f" exitCode=0 Nov 02 17:41:28.955154 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:28.955068 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" event={"ID":"5feaad83-55b3-4503-8eb7-9f5c33f38f9e","Type":"ContainerDied","Data":"9040670e27e394cffeddd06571ea4c78739469d58580bb2a61ff92dfb2aa392f"} Nov 02 17:41:29.026286 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:29.026226 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" Nov 02 17:41:29.170379 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:29.170337 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-maas-api-tls\") pod \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\" (UID: \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\") " Nov 02 17:41:29.170547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:29.170414 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-openshift-service-ca\") pod \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\" (UID: \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\") " Nov 02 17:41:29.170547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:29.170488 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-kube-api-access-pxkd2\" (UniqueName: \"kubernetes.io/projected/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-kube-api-access-pxkd2\") pod \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\" (UID: \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\") " Nov 02 17:41:29.170702 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:29.170573 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-metrics-certs\") pod \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\" (UID: \"5feaad83-55b3-4503-8eb7-9f5c33f38f9e\") " Nov 02 17:41:29.170825 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:29.170790 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-openshift-service-ca" pod "5feaad83-55b3-4503-8eb7-9f5c33f38f9e" (UID: "5feaad83-55b3-4503-8eb7-9f5c33f38f9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:29.170932 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:29.170911 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:29.172537 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:29.172510 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-metrics-certs" pod "5feaad83-55b3-4503-8eb7-9f5c33f38f9e" (UID: "5feaad83-55b3-4503-8eb7-9f5c33f38f9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:29.172677 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:29.172550 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-maas-api-tls" pod "5feaad83-55b3-4503-8eb7-9f5c33f38f9e" (UID: "5feaad83-55b3-4503-8eb7-9f5c33f38f9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:29.172677 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:29.172574 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-kube-api-access-pxkd2" pod "5feaad83-55b3-4503-8eb7-9f5c33f38f9e" (UID: "5feaad83-55b3-4503-8eb7-9f5c33f38f9e"). InnerVolumeSpecName "kube-api-access-pxkd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:29.272161 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:29.272127 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:29.272161 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:29.272161 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxkd2\" (UniqueName: \"kubernetes.io/projected/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-kube-api-access-pxkd2\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:29.272364 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:29.272171 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5feaad83-55b3-4503-8eb7-9f5c33f38f9e-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:29.647992 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:29.647955 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv"] Nov 02 17:41:29.962557 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:29.962528 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" Nov 02 17:41:29.963060 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:29.962553 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv" event={"ID":"5feaad83-55b3-4503-8eb7-9f5c33f38f9e","Type":"ContainerDied","Data":"d23eb7ba9b2f1a43cff54cfea5aca2afeb5361da7e8d09b94a8b10e61326ed3b"} Nov 02 17:41:29.963060 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:29.962634 5601 scope.go:122] "RemoveContainer" containerID="9040670e27e394cffeddd06571ea4c78739469d58580bb2a61ff92dfb2aa392f" Nov 02 17:41:29.980005 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:29.979980 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv"] Nov 02 17:41:29.985717 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:29.985694 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-38f4eea3-74b8b7465f-vzzkv"] Nov 02 17:41:31.425363 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:31.425315 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5feaad83-55b3-4503-8eb7-9f5c33f38f9e" path="/var/lib/kubelet/pods/5feaad83-55b3-4503-8eb7-9f5c33f38f9e/volumes" Nov 02 17:41:32.027395 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.027333 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq" podUID="67ad6740-6d33-412d-8af3-ba486b5848b9" containerName="payload-processing" containerID="cri-o://0b98f31af2d89484b7ca859eac3eab50dac2ac971baabf09722691476d60e6c5" gracePeriod=25 Nov 02 17:41:32.035685 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.035652 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb" podUID="378ef976-7343-4b07-8d8e-9db56e00ae94" containerName="payload-pre-processing" containerID="cri-o://a8f7c744ee33d1912e20f3afafa2b2fc96e31c218a728cc332d72e512a267b87" gracePeriod=25 Nov 02 17:41:32.290186 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.290126 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq" Nov 02 17:41:32.302383 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.302354 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67ad6740-6d33-412d-8af3-ba486b5848b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ad6740-6d33-412d-8af3-ba486b5848b9-plugins-config-volume\") pod \"67ad6740-6d33-412d-8af3-ba486b5848b9\" (UID: \"67ad6740-6d33-412d-8af3-ba486b5848b9\") " Nov 02 17:41:32.302469 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.302431 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ad6740-6d33-412d-8af3-ba486b5848b9-kube-api-access-6bblx\" (UniqueName: \"kubernetes.io/projected/67ad6740-6d33-412d-8af3-ba486b5848b9-kube-api-access-6bblx\") pod \"67ad6740-6d33-412d-8af3-ba486b5848b9\" (UID: \"67ad6740-6d33-412d-8af3-ba486b5848b9\") " Nov 02 17:41:32.302758 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.302734 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ad6740-6d33-412d-8af3-ba486b5848b9-plugins-config-volume" pod "67ad6740-6d33-412d-8af3-ba486b5848b9" (UID: "67ad6740-6d33-412d-8af3-ba486b5848b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:32.304421 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.304400 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ad6740-6d33-412d-8af3-ba486b5848b9-kube-api-access-6bblx" pod "67ad6740-6d33-412d-8af3-ba486b5848b9" (UID: "67ad6740-6d33-412d-8af3-ba486b5848b9"). InnerVolumeSpecName "kube-api-access-6bblx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:32.318446 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.318421 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb" Nov 02 17:41:32.403774 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.403743 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/378ef976-7343-4b07-8d8e-9db56e00ae94-kube-api-access-q7xtq\" (UniqueName: \"kubernetes.io/projected/378ef976-7343-4b07-8d8e-9db56e00ae94-kube-api-access-q7xtq\") pod \"378ef976-7343-4b07-8d8e-9db56e00ae94\" (UID: \"378ef976-7343-4b07-8d8e-9db56e00ae94\") " Nov 02 17:41:32.403919 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.403873 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/378ef976-7343-4b07-8d8e-9db56e00ae94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/378ef976-7343-4b07-8d8e-9db56e00ae94-plugins-config-volume\") pod \"378ef976-7343-4b07-8d8e-9db56e00ae94\" (UID: \"378ef976-7343-4b07-8d8e-9db56e00ae94\") " Nov 02 17:41:32.404051 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.404040 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ad6740-6d33-412d-8af3-ba486b5848b9-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:32.404086 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.404054 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bblx\" (UniqueName: \"kubernetes.io/projected/67ad6740-6d33-412d-8af3-ba486b5848b9-kube-api-access-6bblx\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:32.404263 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.404233 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378ef976-7343-4b07-8d8e-9db56e00ae94-plugins-config-volume" pod "378ef976-7343-4b07-8d8e-9db56e00ae94" (UID: "378ef976-7343-4b07-8d8e-9db56e00ae94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:32.405962 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.405939 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378ef976-7343-4b07-8d8e-9db56e00ae94-kube-api-access-q7xtq" pod "378ef976-7343-4b07-8d8e-9db56e00ae94" (UID: "378ef976-7343-4b07-8d8e-9db56e00ae94"). InnerVolumeSpecName "kube-api-access-q7xtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:32.504970 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.504934 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/378ef976-7343-4b07-8d8e-9db56e00ae94-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:32.504970 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.504960 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7xtq\" (UniqueName: \"kubernetes.io/projected/378ef976-7343-4b07-8d8e-9db56e00ae94-kube-api-access-q7xtq\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:32.981472 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.981439 5601 generic.go:356] "Generic (PLEG): container finished" podID="378ef976-7343-4b07-8d8e-9db56e00ae94" containerID="a8f7c744ee33d1912e20f3afafa2b2fc96e31c218a728cc332d72e512a267b87" exitCode=0 Nov 02 17:41:32.981705 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.981499 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb" Nov 02 17:41:32.981705 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.981525 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb" event={"ID":"378ef976-7343-4b07-8d8e-9db56e00ae94","Type":"ContainerDied","Data":"a8f7c744ee33d1912e20f3afafa2b2fc96e31c218a728cc332d72e512a267b87"} Nov 02 17:41:32.981705 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.981584 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb" event={"ID":"378ef976-7343-4b07-8d8e-9db56e00ae94","Type":"ContainerDied","Data":"897bdd1e5941484bc052e7bb3d1926580ae55865fb785a3f3119ce1eaf056c16"} Nov 02 17:41:32.981705 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.981635 5601 scope.go:122] "RemoveContainer" containerID="a8f7c744ee33d1912e20f3afafa2b2fc96e31c218a728cc332d72e512a267b87" Nov 02 17:41:32.983151 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.983124 5601 generic.go:356] "Generic (PLEG): container finished" podID="67ad6740-6d33-412d-8af3-ba486b5848b9" containerID="0b98f31af2d89484b7ca859eac3eab50dac2ac971baabf09722691476d60e6c5" exitCode=0 Nov 02 17:41:32.983259 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.983156 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq" event={"ID":"67ad6740-6d33-412d-8af3-ba486b5848b9","Type":"ContainerDied","Data":"0b98f31af2d89484b7ca859eac3eab50dac2ac971baabf09722691476d60e6c5"} Nov 02 17:41:32.983259 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.983173 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq" Nov 02 17:41:32.983259 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.983197 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq" event={"ID":"67ad6740-6d33-412d-8af3-ba486b5848b9","Type":"ContainerDied","Data":"1d70a9ad475763e9f1b2ced548eaf82b0fde52cb84e1e51adb6a7bdb75635da3"} Nov 02 17:41:32.990964 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.990947 5601 scope.go:122] "RemoveContainer" containerID="a8f7c744ee33d1912e20f3afafa2b2fc96e31c218a728cc332d72e512a267b87" Nov 02 17:41:32.991274 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:41:32.991253 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f7c744ee33d1912e20f3afafa2b2fc96e31c218a728cc332d72e512a267b87\": container with ID starting with a8f7c744ee33d1912e20f3afafa2b2fc96e31c218a728cc332d72e512a267b87 not found: ID does not exist" containerID="a8f7c744ee33d1912e20f3afafa2b2fc96e31c218a728cc332d72e512a267b87" Nov 02 17:41:32.991360 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.991285 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f7c744ee33d1912e20f3afafa2b2fc96e31c218a728cc332d72e512a267b87"} err="failed to get container status \"a8f7c744ee33d1912e20f3afafa2b2fc96e31c218a728cc332d72e512a267b87\": rpc error: code = NotFound desc = could not find container \"a8f7c744ee33d1912e20f3afafa2b2fc96e31c218a728cc332d72e512a267b87\": container with ID starting with a8f7c744ee33d1912e20f3afafa2b2fc96e31c218a728cc332d72e512a267b87 not found: ID does not exist" Nov 02 17:41:32.991360 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.991308 5601 scope.go:122] "RemoveContainer" containerID="0b98f31af2d89484b7ca859eac3eab50dac2ac971baabf09722691476d60e6c5" Nov 02 17:41:32.999958 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:32.999940 5601 scope.go:122] "RemoveContainer" containerID="0b98f31af2d89484b7ca859eac3eab50dac2ac971baabf09722691476d60e6c5" Nov 02 17:41:33.000230 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:41:33.000215 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b98f31af2d89484b7ca859eac3eab50dac2ac971baabf09722691476d60e6c5\": container with ID starting with 0b98f31af2d89484b7ca859eac3eab50dac2ac971baabf09722691476d60e6c5 not found: ID does not exist" containerID="0b98f31af2d89484b7ca859eac3eab50dac2ac971baabf09722691476d60e6c5" Nov 02 17:41:33.000271 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.000239 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b98f31af2d89484b7ca859eac3eab50dac2ac971baabf09722691476d60e6c5"} err="failed to get container status \"0b98f31af2d89484b7ca859eac3eab50dac2ac971baabf09722691476d60e6c5\": rpc error: code = NotFound desc = could not find container \"0b98f31af2d89484b7ca859eac3eab50dac2ac971baabf09722691476d60e6c5\": container with ID starting with 0b98f31af2d89484b7ca859eac3eab50dac2ac971baabf09722691476d60e6c5 not found: ID does not exist" Nov 02 17:41:33.008777 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.008740 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq"] Nov 02 17:41:33.012051 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.012026 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-38f4eea3-58557b46f5-hvlmq"] Nov 02 17:41:33.025489 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.025455 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb"] Nov 02 17:41:33.030929 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.030905 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-38f4eea3-959664bc9-lrqkb"] Nov 02 17:41:33.101511 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.101479 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm"] Nov 02 17:41:33.145120 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.145071 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf"] Nov 02 17:41:33.145373 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.145342 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" podUID="21d45c50-5921-4206-8fec-1c07a1dd3bef" containerName="maas-api" containerID="cri-o://6c6a08eec52e4b33faafba74f7e1397c0457ecbd1ff0f969d11be9f088085bda" gracePeriod=30 Nov 02 17:41:33.316393 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.316358 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d"] Nov 02 17:41:33.316692 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.316635 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" podUID="5366ed79-2236-4896-8ed6-b34b5d6f4cbc" containerName="maas-api" containerID="cri-o://89984fc515b4a00711ad9b6e14a88d3f57a584c8306699e2308e381446d5bc37" gracePeriod=30 Nov 02 17:41:33.425313 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.425282 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="378ef976-7343-4b07-8d8e-9db56e00ae94" path="/var/lib/kubelet/pods/378ef976-7343-4b07-8d8e-9db56e00ae94/volumes" Nov 02 17:41:33.425682 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.425668 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67ad6740-6d33-412d-8af3-ba486b5848b9" path="/var/lib/kubelet/pods/67ad6740-6d33-412d-8af3-ba486b5848b9/volumes" Nov 02 17:41:33.443511 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.443491 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" Nov 02 17:41:33.518348 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.518312 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d45c50-5921-4206-8fec-1c07a1dd3bef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21d45c50-5921-4206-8fec-1c07a1dd3bef-openshift-service-ca\") pod \"21d45c50-5921-4206-8fec-1c07a1dd3bef\" (UID: \"21d45c50-5921-4206-8fec-1c07a1dd3bef\") " Nov 02 17:41:33.518809 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.518397 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21d45c50-5921-4206-8fec-1c07a1dd3bef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21d45c50-5921-4206-8fec-1c07a1dd3bef-maas-api-tls\") pod \"21d45c50-5921-4206-8fec-1c07a1dd3bef\" (UID: \"21d45c50-5921-4206-8fec-1c07a1dd3bef\") " Nov 02 17:41:33.518809 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.518451 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21d45c50-5921-4206-8fec-1c07a1dd3bef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21d45c50-5921-4206-8fec-1c07a1dd3bef-metrics-certs\") pod \"21d45c50-5921-4206-8fec-1c07a1dd3bef\" (UID: \"21d45c50-5921-4206-8fec-1c07a1dd3bef\") " Nov 02 17:41:33.518809 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.518518 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d45c50-5921-4206-8fec-1c07a1dd3bef-kube-api-access-lcdl6\" (UniqueName: \"kubernetes.io/projected/21d45c50-5921-4206-8fec-1c07a1dd3bef-kube-api-access-lcdl6\") pod \"21d45c50-5921-4206-8fec-1c07a1dd3bef\" (UID: \"21d45c50-5921-4206-8fec-1c07a1dd3bef\") " Nov 02 17:41:33.518985 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.518834 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d45c50-5921-4206-8fec-1c07a1dd3bef-openshift-service-ca" pod "21d45c50-5921-4206-8fec-1c07a1dd3bef" (UID: "21d45c50-5921-4206-8fec-1c07a1dd3bef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:33.523911 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.523860 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d45c50-5921-4206-8fec-1c07a1dd3bef-maas-api-tls" pod "21d45c50-5921-4206-8fec-1c07a1dd3bef" (UID: "21d45c50-5921-4206-8fec-1c07a1dd3bef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:33.525417 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.525381 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d45c50-5921-4206-8fec-1c07a1dd3bef-metrics-certs" pod "21d45c50-5921-4206-8fec-1c07a1dd3bef" (UID: "21d45c50-5921-4206-8fec-1c07a1dd3bef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:33.525522 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.525422 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d45c50-5921-4206-8fec-1c07a1dd3bef-kube-api-access-lcdl6" pod "21d45c50-5921-4206-8fec-1c07a1dd3bef" (UID: "21d45c50-5921-4206-8fec-1c07a1dd3bef"). InnerVolumeSpecName "kube-api-access-lcdl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:33.589796 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.589767 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" Nov 02 17:41:33.619387 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.619359 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-maas-api-tls\") pod \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\" (UID: \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\") " Nov 02 17:41:33.619562 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.619416 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-kube-api-access-7wwqc\" (UniqueName: \"kubernetes.io/projected/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-kube-api-access-7wwqc\") pod \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\" (UID: \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\") " Nov 02 17:41:33.619562 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.619446 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-metrics-certs\") pod \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\" (UID: \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\") " Nov 02 17:41:33.619562 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.619483 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-openshift-service-ca\") pod \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\" (UID: \"5366ed79-2236-4896-8ed6-b34b5d6f4cbc\") " Nov 02 17:41:33.619850 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.619827 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21d45c50-5921-4206-8fec-1c07a1dd3bef-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:33.619910 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.619851 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21d45c50-5921-4206-8fec-1c07a1dd3bef-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:33.619910 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.619866 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21d45c50-5921-4206-8fec-1c07a1dd3bef-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:33.619910 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.619880 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcdl6\" (UniqueName: \"kubernetes.io/projected/21d45c50-5921-4206-8fec-1c07a1dd3bef-kube-api-access-lcdl6\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:33.620256 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.620159 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-openshift-service-ca" pod "5366ed79-2236-4896-8ed6-b34b5d6f4cbc" (UID: "5366ed79-2236-4896-8ed6-b34b5d6f4cbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:33.621911 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.621884 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-metrics-certs" pod "5366ed79-2236-4896-8ed6-b34b5d6f4cbc" (UID: "5366ed79-2236-4896-8ed6-b34b5d6f4cbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:33.622323 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.622291 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-maas-api-tls" pod "5366ed79-2236-4896-8ed6-b34b5d6f4cbc" (UID: "5366ed79-2236-4896-8ed6-b34b5d6f4cbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:33.622630 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.622598 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-kube-api-access-7wwqc" pod "5366ed79-2236-4896-8ed6-b34b5d6f4cbc" (UID: "5366ed79-2236-4896-8ed6-b34b5d6f4cbc"). InnerVolumeSpecName "kube-api-access-7wwqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:33.720937 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.720901 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:33.720937 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.720931 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wwqc\" (UniqueName: \"kubernetes.io/projected/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-kube-api-access-7wwqc\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:33.720937 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.720942 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:33.721259 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.720950 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5366ed79-2236-4896-8ed6-b34b5d6f4cbc-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:33.992818 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.992786 5601 generic.go:356] "Generic (PLEG): container finished" podID="21d45c50-5921-4206-8fec-1c07a1dd3bef" containerID="6c6a08eec52e4b33faafba74f7e1397c0457ecbd1ff0f969d11be9f088085bda" exitCode=0 Nov 02 17:41:33.992994 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.992931 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" Nov 02 17:41:33.994636 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.994595 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" event={"ID":"21d45c50-5921-4206-8fec-1c07a1dd3bef","Type":"ContainerDied","Data":"6c6a08eec52e4b33faafba74f7e1397c0457ecbd1ff0f969d11be9f088085bda"} Nov 02 17:41:33.994751 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.994659 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf" event={"ID":"21d45c50-5921-4206-8fec-1c07a1dd3bef","Type":"ContainerDied","Data":"431c141d41e673e944d68d07fc494ac842c7f4c2d7bdd4ae9b9e216e15f354f3"} Nov 02 17:41:33.994751 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.994683 5601 scope.go:122] "RemoveContainer" containerID="6c6a08eec52e4b33faafba74f7e1397c0457ecbd1ff0f969d11be9f088085bda" Nov 02 17:41:33.997350 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.997321 5601 generic.go:356] "Generic (PLEG): container finished" podID="5366ed79-2236-4896-8ed6-b34b5d6f4cbc" containerID="89984fc515b4a00711ad9b6e14a88d3f57a584c8306699e2308e381446d5bc37" exitCode=0 Nov 02 17:41:33.997477 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.997377 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" Nov 02 17:41:33.997477 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.997416 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" event={"ID":"5366ed79-2236-4896-8ed6-b34b5d6f4cbc","Type":"ContainerDied","Data":"89984fc515b4a00711ad9b6e14a88d3f57a584c8306699e2308e381446d5bc37"} Nov 02 17:41:33.997477 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:33.997457 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d" event={"ID":"5366ed79-2236-4896-8ed6-b34b5d6f4cbc","Type":"ContainerDied","Data":"39a2be6b037b367a581bdb4eaf8ce09bdd0303bde090ec5cb1b93da3b141e196"} Nov 02 17:41:34.011580 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:34.011549 5601 scope.go:122] "RemoveContainer" containerID="6c6a08eec52e4b33faafba74f7e1397c0457ecbd1ff0f969d11be9f088085bda" Nov 02 17:41:34.012081 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:41:34.012054 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6a08eec52e4b33faafba74f7e1397c0457ecbd1ff0f969d11be9f088085bda\": container with ID starting with 6c6a08eec52e4b33faafba74f7e1397c0457ecbd1ff0f969d11be9f088085bda not found: ID does not exist" containerID="6c6a08eec52e4b33faafba74f7e1397c0457ecbd1ff0f969d11be9f088085bda" Nov 02 17:41:34.012184 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:34.012086 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6a08eec52e4b33faafba74f7e1397c0457ecbd1ff0f969d11be9f088085bda"} err="failed to get container status \"6c6a08eec52e4b33faafba74f7e1397c0457ecbd1ff0f969d11be9f088085bda\": rpc error: code = NotFound desc = could not find container \"6c6a08eec52e4b33faafba74f7e1397c0457ecbd1ff0f969d11be9f088085bda\": container with ID starting with 6c6a08eec52e4b33faafba74f7e1397c0457ecbd1ff0f969d11be9f088085bda not found: ID does not exist" Nov 02 17:41:34.012184 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:34.012104 5601 scope.go:122] "RemoveContainer" containerID="89984fc515b4a00711ad9b6e14a88d3f57a584c8306699e2308e381446d5bc37" Nov 02 17:41:34.020842 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:34.020818 5601 scope.go:122] "RemoveContainer" containerID="89984fc515b4a00711ad9b6e14a88d3f57a584c8306699e2308e381446d5bc37" Nov 02 17:41:34.021138 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:41:34.021119 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89984fc515b4a00711ad9b6e14a88d3f57a584c8306699e2308e381446d5bc37\": container with ID starting with 89984fc515b4a00711ad9b6e14a88d3f57a584c8306699e2308e381446d5bc37 not found: ID does not exist" containerID="89984fc515b4a00711ad9b6e14a88d3f57a584c8306699e2308e381446d5bc37" Nov 02 17:41:34.021202 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:34.021150 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89984fc515b4a00711ad9b6e14a88d3f57a584c8306699e2308e381446d5bc37"} err="failed to get container status \"89984fc515b4a00711ad9b6e14a88d3f57a584c8306699e2308e381446d5bc37\": rpc error: code = NotFound desc = could not find container \"89984fc515b4a00711ad9b6e14a88d3f57a584c8306699e2308e381446d5bc37\": container with ID starting with 89984fc515b4a00711ad9b6e14a88d3f57a584c8306699e2308e381446d5bc37 not found: ID does not exist" Nov 02 17:41:34.029644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:34.029593 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf"] Nov 02 17:41:34.031225 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:34.031203 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0ae9c41e-866fb87b9f-km7wf"] Nov 02 17:41:34.051629 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:34.051586 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d"] Nov 02 17:41:34.056782 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:34.056753 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-868a2ee9-779dbfdcfc-v6p5d"] Nov 02 17:41:34.484799 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:41:34.484760 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:41:34.649044 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:34.649002 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv" podUID="c04c7b10-dac8-4ba7-a86b-801eec1f95fd" containerName="payload-pre-processing" containerID="cri-o://7fcaa9187dd24bc29a12e01297089dc0cf027a5ae663f06a9bb3adcf4bb79b57" gracePeriod=25 Nov 02 17:41:34.688430 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:34.688398 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd"] Nov 02 17:41:34.688737 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:34.688686 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" podUID="4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782" containerName="maas-api" containerID="cri-o://0a631347c644bc1ed69b88cbdcb24629f5699885501f674b6ecd29dc714775f3" gracePeriod=30 Nov 02 17:41:34.951112 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:34.951080 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv" Nov 02 17:41:34.966605 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:34.966577 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" Nov 02 17:41:35.014372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.014327 5601 generic.go:356] "Generic (PLEG): container finished" podID="4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782" containerID="0a631347c644bc1ed69b88cbdcb24629f5699885501f674b6ecd29dc714775f3" exitCode=0 Nov 02 17:41:35.014529 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.014513 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" event={"ID":"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782","Type":"ContainerDied","Data":"0a631347c644bc1ed69b88cbdcb24629f5699885501f674b6ecd29dc714775f3"} Nov 02 17:41:35.014661 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.014609 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" event={"ID":"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782","Type":"ContainerDied","Data":"724bc2b81e92b907f087144382865bda5346f6350d08e34b07d992719e9ffb38"} Nov 02 17:41:35.014745 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.014695 5601 scope.go:122] "RemoveContainer" containerID="0a631347c644bc1ed69b88cbdcb24629f5699885501f674b6ecd29dc714775f3" Nov 02 17:41:35.014923 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.014892 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd" Nov 02 17:41:35.020502 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.020472 5601 generic.go:356] "Generic (PLEG): container finished" podID="c04c7b10-dac8-4ba7-a86b-801eec1f95fd" containerID="7fcaa9187dd24bc29a12e01297089dc0cf027a5ae663f06a9bb3adcf4bb79b57" exitCode=0 Nov 02 17:41:35.020650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.020570 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv" event={"ID":"c04c7b10-dac8-4ba7-a86b-801eec1f95fd","Type":"ContainerDied","Data":"7fcaa9187dd24bc29a12e01297089dc0cf027a5ae663f06a9bb3adcf4bb79b57"} Nov 02 17:41:35.020730 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.020653 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv" event={"ID":"c04c7b10-dac8-4ba7-a86b-801eec1f95fd","Type":"ContainerDied","Data":"0b4e061daecf36eb48777500c4d1b560c6d6664edf8c4c5207c04721779d6a32"} Nov 02 17:41:35.020730 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.020572 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv" Nov 02 17:41:35.028446 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.028421 5601 scope.go:122] "RemoveContainer" containerID="0a631347c644bc1ed69b88cbdcb24629f5699885501f674b6ecd29dc714775f3" Nov 02 17:41:35.028846 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:41:35.028817 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a631347c644bc1ed69b88cbdcb24629f5699885501f674b6ecd29dc714775f3\": container with ID starting with 0a631347c644bc1ed69b88cbdcb24629f5699885501f674b6ecd29dc714775f3 not found: ID does not exist" containerID="0a631347c644bc1ed69b88cbdcb24629f5699885501f674b6ecd29dc714775f3" Nov 02 17:41:35.028928 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.028889 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a631347c644bc1ed69b88cbdcb24629f5699885501f674b6ecd29dc714775f3"} err="failed to get container status \"0a631347c644bc1ed69b88cbdcb24629f5699885501f674b6ecd29dc714775f3\": rpc error: code = NotFound desc = could not find container \"0a631347c644bc1ed69b88cbdcb24629f5699885501f674b6ecd29dc714775f3\": container with ID starting with 0a631347c644bc1ed69b88cbdcb24629f5699885501f674b6ecd29dc714775f3 not found: ID does not exist" Nov 02 17:41:35.028987 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.028934 5601 scope.go:122] "RemoveContainer" containerID="7fcaa9187dd24bc29a12e01297089dc0cf027a5ae663f06a9bb3adcf4bb79b57" Nov 02 17:41:35.032535 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.032503 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c04c7b10-dac8-4ba7-a86b-801eec1f95fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04c7b10-dac8-4ba7-a86b-801eec1f95fd-plugins-config-volume\") pod \"c04c7b10-dac8-4ba7-a86b-801eec1f95fd\" (UID: \"c04c7b10-dac8-4ba7-a86b-801eec1f95fd\") " Nov 02 17:41:35.032647 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.032582 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-maas-api-tls\") pod \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\" (UID: \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\") " Nov 02 17:41:35.032647 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.032634 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-openshift-service-ca\") pod \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\" (UID: \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\") " Nov 02 17:41:35.032765 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.032679 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04c7b10-dac8-4ba7-a86b-801eec1f95fd-kube-api-access-fmm4r\" (UniqueName: \"kubernetes.io/projected/c04c7b10-dac8-4ba7-a86b-801eec1f95fd-kube-api-access-fmm4r\") pod \"c04c7b10-dac8-4ba7-a86b-801eec1f95fd\" (UID: \"c04c7b10-dac8-4ba7-a86b-801eec1f95fd\") " Nov 02 17:41:35.032765 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.032720 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-kube-api-access-k227m\" (UniqueName: \"kubernetes.io/projected/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-kube-api-access-k227m\") pod \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\" (UID: \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\") " Nov 02 17:41:35.032862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.032765 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-metrics-certs\") pod \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\" (UID: \"4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782\") " Nov 02 17:41:35.033281 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.033234 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04c7b10-dac8-4ba7-a86b-801eec1f95fd-plugins-config-volume" pod "c04c7b10-dac8-4ba7-a86b-801eec1f95fd" (UID: "c04c7b10-dac8-4ba7-a86b-801eec1f95fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:35.033951 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.033929 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-openshift-service-ca" pod "4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782" (UID: "4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:35.035138 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.035120 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-maas-api-tls" pod "4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782" (UID: "4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:35.035830 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.035804 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-metrics-certs" pod "4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782" (UID: "4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:35.036795 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.036765 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-kube-api-access-k227m" pod "4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782" (UID: "4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782"). InnerVolumeSpecName "kube-api-access-k227m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:35.038543 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.038367 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04c7b10-dac8-4ba7-a86b-801eec1f95fd-kube-api-access-fmm4r" pod "c04c7b10-dac8-4ba7-a86b-801eec1f95fd" (UID: "c04c7b10-dac8-4ba7-a86b-801eec1f95fd"). InnerVolumeSpecName "kube-api-access-fmm4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:35.041336 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.041307 5601 scope.go:122] "RemoveContainer" containerID="7fcaa9187dd24bc29a12e01297089dc0cf027a5ae663f06a9bb3adcf4bb79b57" Nov 02 17:41:35.041940 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:41:35.041834 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fcaa9187dd24bc29a12e01297089dc0cf027a5ae663f06a9bb3adcf4bb79b57\": container with ID starting with 7fcaa9187dd24bc29a12e01297089dc0cf027a5ae663f06a9bb3adcf4bb79b57 not found: ID does not exist" containerID="7fcaa9187dd24bc29a12e01297089dc0cf027a5ae663f06a9bb3adcf4bb79b57" Nov 02 17:41:35.042047 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.041956 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fcaa9187dd24bc29a12e01297089dc0cf027a5ae663f06a9bb3adcf4bb79b57"} err="failed to get container status \"7fcaa9187dd24bc29a12e01297089dc0cf027a5ae663f06a9bb3adcf4bb79b57\": rpc error: code = NotFound desc = could not find container \"7fcaa9187dd24bc29a12e01297089dc0cf027a5ae663f06a9bb3adcf4bb79b57\": container with ID starting with 7fcaa9187dd24bc29a12e01297089dc0cf027a5ae663f06a9bb3adcf4bb79b57 not found: ID does not exist" Nov 02 17:41:35.113038 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.112953 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl"] Nov 02 17:41:35.113353 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113340 5601 cpu_manager.go:414] "containerMap: removing container" podUID="67ad6740-6d33-412d-8af3-ba486b5848b9" containerName="payload-processing" Nov 02 17:41:35.113405 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113355 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67ad6740-6d33-412d-8af3-ba486b5848b9" containerName="payload-processing" Nov 02 17:41:35.113405 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113371 5601 cpu_manager.go:414] "containerMap: removing container" podUID="378ef976-7343-4b07-8d8e-9db56e00ae94" containerName="payload-pre-processing" Nov 02 17:41:35.113405 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113377 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="378ef976-7343-4b07-8d8e-9db56e00ae94" containerName="payload-pre-processing" Nov 02 17:41:35.113405 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113388 5601 cpu_manager.go:414] "containerMap: removing container" podUID="21d45c50-5921-4206-8fec-1c07a1dd3bef" containerName="maas-api" Nov 02 17:41:35.113405 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113393 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d45c50-5921-4206-8fec-1c07a1dd3bef" containerName="maas-api" Nov 02 17:41:35.113405 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113400 5601 cpu_manager.go:414] "containerMap: removing container" podUID="5366ed79-2236-4896-8ed6-b34b5d6f4cbc" containerName="maas-api" Nov 02 17:41:35.113405 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113405 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5366ed79-2236-4896-8ed6-b34b5d6f4cbc" containerName="maas-api" Nov 02 17:41:35.113659 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113421 5601 cpu_manager.go:414] "containerMap: removing container" podUID="4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782" containerName="maas-api" Nov 02 17:41:35.113659 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113426 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782" containerName="maas-api" Nov 02 17:41:35.113659 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113455 5601 cpu_manager.go:414] "containerMap: removing container" podUID="c04c7b10-dac8-4ba7-a86b-801eec1f95fd" containerName="payload-pre-processing" Nov 02 17:41:35.113659 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113463 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04c7b10-dac8-4ba7-a86b-801eec1f95fd" containerName="payload-pre-processing" Nov 02 17:41:35.113659 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113476 5601 cpu_manager.go:414] "containerMap: removing container" podUID="5feaad83-55b3-4503-8eb7-9f5c33f38f9e" containerName="maas-api" Nov 02 17:41:35.113659 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113481 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5feaad83-55b3-4503-8eb7-9f5c33f38f9e" containerName="maas-api" Nov 02 17:41:35.113659 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113552 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="67ad6740-6d33-412d-8af3-ba486b5848b9" containerName="payload-processing" Nov 02 17:41:35.113659 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113560 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="378ef976-7343-4b07-8d8e-9db56e00ae94" containerName="payload-pre-processing" Nov 02 17:41:35.113659 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113569 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d45c50-5921-4206-8fec-1c07a1dd3bef" containerName="maas-api" Nov 02 17:41:35.113659 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113575 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="5366ed79-2236-4896-8ed6-b34b5d6f4cbc" containerName="maas-api" Nov 02 17:41:35.113659 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113582 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782" containerName="maas-api" Nov 02 17:41:35.113659 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113587 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04c7b10-dac8-4ba7-a86b-801eec1f95fd" containerName="payload-pre-processing" Nov 02 17:41:35.113659 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.113594 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="5feaad83-55b3-4503-8eb7-9f5c33f38f9e" containerName="maas-api" Nov 02 17:41:35.119059 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.119031 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.125420 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.125394 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-66eed1-openshift-default-dockercfg-dd2rz\"" Nov 02 17:41:35.134302 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.134268 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04c7b10-dac8-4ba7-a86b-801eec1f95fd-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:35.134302 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.134301 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:35.134476 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.134316 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:35.134476 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.134330 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmm4r\" (UniqueName: \"kubernetes.io/projected/c04c7b10-dac8-4ba7-a86b-801eec1f95fd-kube-api-access-fmm4r\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:35.134476 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.134342 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k227m\" (UniqueName: \"kubernetes.io/projected/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-kube-api-access-k227m\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:35.134476 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.134355 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:35.152496 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.152457 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl"] Nov 02 17:41:35.235770 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.235731 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.235770 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.235774 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r6sn\" (UniqueName: \"kubernetes.io/projected/f26c04ee-0060-4863-b50f-2eed99aad8b1-kube-api-access-5r6sn\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.236031 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.235875 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.236031 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.235900 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.236031 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.235923 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.236031 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.235952 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26c04ee-0060-4863-b50f-2eed99aad8b1-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.236252 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.236033 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.236252 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.236171 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.236252 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.236205 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.337444 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.337411 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.337594 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.337464 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.337594 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.337497 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.337594 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.337518 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r6sn\" (UniqueName: \"kubernetes.io/projected/f26c04ee-0060-4863-b50f-2eed99aad8b1-kube-api-access-5r6sn\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.337594 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.337556 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.337594 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.337590 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.337972 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.337656 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.337972 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.337700 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26c04ee-0060-4863-b50f-2eed99aad8b1-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.337972 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.337725 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.338152 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.337984 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.338152 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.338006 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.338152 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.338081 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.338763 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.338268 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.338763 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.338663 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26c04ee-0060-4863-b50f-2eed99aad8b1-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.340397 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.340372 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.340503 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.340401 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.355907 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.355832 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd"] Nov 02 17:41:35.369286 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.369211 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.369419 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.369310 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r6sn\" (UniqueName: \"kubernetes.io/projected/f26c04ee-0060-4863-b50f-2eed99aad8b1-kube-api-access-5r6sn\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.372182 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.372149 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-55f9886f-7bb4496d7c-nlcfd"] Nov 02 17:41:35.393202 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.393166 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv"] Nov 02 17:41:35.401113 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.401076 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ae9c41e-676dd647bc-kglrv"] Nov 02 17:41:35.407338 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.407307 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz"] Nov 02 17:41:35.412749 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.412718 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.418449 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.418423 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-4b97c6-openshift-default-dockercfg-smxvn\"" Nov 02 17:41:35.425873 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.425845 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21d45c50-5921-4206-8fec-1c07a1dd3bef" path="/var/lib/kubelet/pods/21d45c50-5921-4206-8fec-1c07a1dd3bef/volumes" Nov 02 17:41:35.426266 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.426248 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782" path="/var/lib/kubelet/pods/4e2cd242-0c2f-44bb-85e6-cf5a3a6d2782/volumes" Nov 02 17:41:35.426768 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.426750 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5366ed79-2236-4896-8ed6-b34b5d6f4cbc" path="/var/lib/kubelet/pods/5366ed79-2236-4896-8ed6-b34b5d6f4cbc/volumes" Nov 02 17:41:35.427411 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.427393 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c04c7b10-dac8-4ba7-a86b-801eec1f95fd" path="/var/lib/kubelet/pods/c04c7b10-dac8-4ba7-a86b-801eec1f95fd/volumes" Nov 02 17:41:35.431327 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.431299 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz"] Nov 02 17:41:35.434364 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.434343 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:35.540135 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.540091 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.540316 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.540149 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.540316 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.540214 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.540316 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.540236 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.540316 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.540272 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.540316 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.540299 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.540577 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.540332 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.540577 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.540362 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.540577 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.540390 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9pvk\" (UniqueName: \"kubernetes.io/projected/db8e57a9-62e6-4229-921d-f5a40cde2a7a-kube-api-access-j9pvk\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.641666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.640927 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.641666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.640980 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.641666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.641024 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.641666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.641056 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.641666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.641086 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.641666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.641115 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.641666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.641144 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9pvk\" (UniqueName: \"kubernetes.io/projected/db8e57a9-62e6-4229-921d-f5a40cde2a7a-kube-api-access-j9pvk\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.641666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.641246 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.641666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.641271 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.641666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.641281 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.642339 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.641961 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.642339 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.642227 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.642539 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.642513 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.642929 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.642901 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.645104 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.645056 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.645289 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.645269 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.651343 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.651309 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9pvk\" (UniqueName: \"kubernetes.io/projected/db8e57a9-62e6-4229-921d-f5a40cde2a7a-kube-api-access-j9pvk\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.652091 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.652067 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.674884 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.674850 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg"] Nov 02 17:41:35.679920 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.679896 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" Nov 02 17:41:35.692345 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.692313 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg"] Nov 02 17:41:35.728861 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.728823 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-a37cf0\"" Nov 02 17:41:35.728861 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.728825 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-a37cf0-dockercfg-mpbmh\"" Nov 02 17:41:35.731259 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.729991 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:35.743015 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.742978 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bdbq\" (UniqueName: \"kubernetes.io/projected/28611469-b9c9-435a-87e1-1b7b37f0c5ff-kube-api-access-4bdbq\") pod \"payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg\" (UID: \"28611469-b9c9-435a-87e1-1b7b37f0c5ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" Nov 02 17:41:35.743334 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.743060 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28611469-b9c9-435a-87e1-1b7b37f0c5ff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg\" (UID: \"28611469-b9c9-435a-87e1-1b7b37f0c5ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" Nov 02 17:41:35.744106 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.744034 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl"] Nov 02 17:41:35.753328 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:41:35.753286 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26c04ee_0060_4863_b50f_2eed99aad8b1.slice/crio-90322c0dabf0d9cc4407a6e750fb863d9b20ab3fbc5719975f1ca4dd6e1fe938 WatchSource:0}: Error finding container 90322c0dabf0d9cc4407a6e750fb863d9b20ab3fbc5719975f1ca4dd6e1fe938: Status 404 returned error can't find the container with id 90322c0dabf0d9cc4407a6e750fb863d9b20ab3fbc5719975f1ca4dd6e1fe938 Nov 02 17:41:35.757048 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.757000 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:35.757160 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.757090 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:35.757160 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.757141 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:35.844887 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.844847 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bdbq\" (UniqueName: \"kubernetes.io/projected/28611469-b9c9-435a-87e1-1b7b37f0c5ff-kube-api-access-4bdbq\") pod \"payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg\" (UID: \"28611469-b9c9-435a-87e1-1b7b37f0c5ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" Nov 02 17:41:35.845129 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.844940 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28611469-b9c9-435a-87e1-1b7b37f0c5ff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg\" (UID: \"28611469-b9c9-435a-87e1-1b7b37f0c5ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" Nov 02 17:41:35.846329 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.846294 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28611469-b9c9-435a-87e1-1b7b37f0c5ff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg\" (UID: \"28611469-b9c9-435a-87e1-1b7b37f0c5ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" Nov 02 17:41:35.855304 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.855277 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bdbq\" (UniqueName: \"kubernetes.io/projected/28611469-b9c9-435a-87e1-1b7b37f0c5ff-kube-api-access-4bdbq\") pod \"payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg\" (UID: \"28611469-b9c9-435a-87e1-1b7b37f0c5ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" Nov 02 17:41:35.994407 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.994363 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677"] Nov 02 17:41:35.998910 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:35.998877 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" Nov 02 17:41:36.000285 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.000261 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" Nov 02 17:41:36.009649 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.008933 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677"] Nov 02 17:41:36.032208 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.032175 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz"] Nov 02 17:41:36.032419 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.032330 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-a37cf0-serving-cert\"" Nov 02 17:41:36.032482 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.032437 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tqgzn\"" Nov 02 17:41:36.032601 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.032561 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 17:41:36.037169 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:41:36.037140 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8e57a9_62e6_4229_921d_f5a40cde2a7a.slice/crio-1f23bb7315cb0832f8e76d4d1276dcc6f86f755c7d1064cef8a8bbad182b8999 WatchSource:0}: Error finding container 1f23bb7315cb0832f8e76d4d1276dcc6f86f755c7d1064cef8a8bbad182b8999: Status 404 returned error can't find the container with id 1f23bb7315cb0832f8e76d4d1276dcc6f86f755c7d1064cef8a8bbad182b8999 Nov 02 17:41:36.037470 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.037437 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" event={"ID":"f26c04ee-0060-4863-b50f-2eed99aad8b1","Type":"ContainerStarted","Data":"857d53115e96c7bc173b8372ff4b8b85ab10357ee05cb82a85f0c29bc5b55d53"} Nov 02 17:41:36.037551 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.037509 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" event={"ID":"f26c04ee-0060-4863-b50f-2eed99aad8b1","Type":"ContainerStarted","Data":"90322c0dabf0d9cc4407a6e750fb863d9b20ab3fbc5719975f1ca4dd6e1fe938"} Nov 02 17:41:36.038916 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.038882 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:36.039012 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.038964 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:36.039063 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.039010 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:36.048086 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.048040 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b0271e7-cb07-45b2-a349-32f49ce6e442-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677\" (UID: \"1b0271e7-cb07-45b2-a349-32f49ce6e442\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" Nov 02 17:41:36.048281 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.048094 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b0271e7-cb07-45b2-a349-32f49ce6e442-metrics-certs\") pod \"maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677\" (UID: \"1b0271e7-cb07-45b2-a349-32f49ce6e442\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" Nov 02 17:41:36.048281 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.048169 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0271e7-cb07-45b2-a349-32f49ce6e442-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677\" (UID: \"1b0271e7-cb07-45b2-a349-32f49ce6e442\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" Nov 02 17:41:36.048281 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.048197 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx42v\" (UniqueName: \"kubernetes.io/projected/1b0271e7-cb07-45b2-a349-32f49ce6e442-kube-api-access-qx42v\") pod \"maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677\" (UID: \"1b0271e7-cb07-45b2-a349-32f49ce6e442\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" Nov 02 17:41:36.062656 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.060183 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" podStartSLOduration=1.060164621 podStartE2EDuration="1.060164621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:36.059686746 +0000 UTC m=+460.750740760" watchObservedRunningTime="2026-11-02 17:41:36.060164621 +0000 UTC m=+460.751218634" Nov 02 17:41:36.149795 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.149754 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b0271e7-cb07-45b2-a349-32f49ce6e442-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677\" (UID: \"1b0271e7-cb07-45b2-a349-32f49ce6e442\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" Nov 02 17:41:36.149965 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.149834 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b0271e7-cb07-45b2-a349-32f49ce6e442-metrics-certs\") pod \"maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677\" (UID: \"1b0271e7-cb07-45b2-a349-32f49ce6e442\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" Nov 02 17:41:36.150165 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.150140 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0271e7-cb07-45b2-a349-32f49ce6e442-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677\" (UID: \"1b0271e7-cb07-45b2-a349-32f49ce6e442\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" Nov 02 17:41:36.150315 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.150294 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx42v\" (UniqueName: \"kubernetes.io/projected/1b0271e7-cb07-45b2-a349-32f49ce6e442-kube-api-access-qx42v\") pod \"maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677\" (UID: \"1b0271e7-cb07-45b2-a349-32f49ce6e442\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" Nov 02 17:41:36.150914 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.150891 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0271e7-cb07-45b2-a349-32f49ce6e442-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677\" (UID: \"1b0271e7-cb07-45b2-a349-32f49ce6e442\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" Nov 02 17:41:36.152308 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.152290 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b0271e7-cb07-45b2-a349-32f49ce6e442-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677\" (UID: \"1b0271e7-cb07-45b2-a349-32f49ce6e442\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" Nov 02 17:41:36.152712 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.152679 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b0271e7-cb07-45b2-a349-32f49ce6e442-metrics-certs\") pod \"maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677\" (UID: \"1b0271e7-cb07-45b2-a349-32f49ce6e442\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" Nov 02 17:41:36.161656 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.161628 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx42v\" (UniqueName: \"kubernetes.io/projected/1b0271e7-cb07-45b2-a349-32f49ce6e442-kube-api-access-qx42v\") pod \"maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677\" (UID: \"1b0271e7-cb07-45b2-a349-32f49ce6e442\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" Nov 02 17:41:36.295757 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.295730 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg"] Nov 02 17:41:36.298887 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:41:36.298851 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28611469_b9c9_435a_87e1_1b7b37f0c5ff.slice/crio-c7b686de7493d2b6388a1fd22a697511592e38b6710ac344b7312043e818bb64 WatchSource:0}: Error finding container c7b686de7493d2b6388a1fd22a697511592e38b6710ac344b7312043e818bb64: Status 404 returned error can't find the container with id c7b686de7493d2b6388a1fd22a697511592e38b6710ac344b7312043e818bb64 Nov 02 17:41:36.330514 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.330484 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" Nov 02 17:41:36.435428 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.435389 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:36.441029 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.440996 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:36.627455 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:36.627014 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677"] Nov 02 17:41:37.045523 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:37.045483 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" event={"ID":"28611469-b9c9-435a-87e1-1b7b37f0c5ff","Type":"ContainerStarted","Data":"5c583e5433b1673b7c0df28429c0344587a791275595b4d4588e57ac24ddfd08"} Nov 02 17:41:37.046017 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:37.045554 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" event={"ID":"28611469-b9c9-435a-87e1-1b7b37f0c5ff","Type":"ContainerStarted","Data":"c7b686de7493d2b6388a1fd22a697511592e38b6710ac344b7312043e818bb64"} Nov 02 17:41:37.046017 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:37.045767 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" Nov 02 17:41:37.047910 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:37.047880 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" event={"ID":"db8e57a9-62e6-4229-921d-f5a40cde2a7a","Type":"ContainerStarted","Data":"e3816cb714ea3f3f23240adae24bafd276db6a0a51b6df93dd5432bf1b7d1981"} Nov 02 17:41:37.048053 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:37.047927 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" event={"ID":"db8e57a9-62e6-4229-921d-f5a40cde2a7a","Type":"ContainerStarted","Data":"1f23bb7315cb0832f8e76d4d1276dcc6f86f755c7d1064cef8a8bbad182b8999"} Nov 02 17:41:37.049425 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:37.049393 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" event={"ID":"1b0271e7-cb07-45b2-a349-32f49ce6e442","Type":"ContainerStarted","Data":"e24c0ac9637e8cd3d6f90483b207baecaba270e40089a291311ea0e86b000679"} Nov 02 17:41:37.049657 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:37.049640 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:37.050513 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:37.050497 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:37.063316 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:37.063276 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" podStartSLOduration=2.063263225 podStartE2EDuration="2.063263225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:37.062904468 +0000 UTC m=+461.753958481" watchObservedRunningTime="2026-11-02 17:41:37.063263225 +0000 UTC m=+461.754317237" Nov 02 17:41:37.083360 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:37.083309 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" podStartSLOduration=2.083292256 podStartE2EDuration="2.083292256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:37.08320508 +0000 UTC m=+461.774259095" watchObservedRunningTime="2026-11-02 17:41:37.083292256 +0000 UTC m=+461.774346273" Nov 02 17:41:37.730894 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:37.730852 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:37.736289 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:37.736248 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:38.056914 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:38.056826 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" event={"ID":"1b0271e7-cb07-45b2-a349-32f49ce6e442","Type":"ContainerStarted","Data":"70f34271d75fae001ba2b1111b63b47cd40a3f107fb97483e0961a40c238e597"} Nov 02 17:41:38.057679 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:38.057650 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" Nov 02 17:41:38.057806 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:38.057691 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:38.058797 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:38.058768 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:38.101841 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:38.101802 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm" podUID="ea9816f4-bad9-4622-a16c-6f7846f72c9a" containerName="payload-processing" containerID="cri-o://f5132738363890611778c6e359e4b6472cbea25197dc1574a2b87f6d7b7962e6" gracePeriod=25 Nov 02 17:41:38.102838 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:38.102795 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" podStartSLOduration=2.600678768 podStartE2EDuration="3.10278216s" totalImagesPullingTime="502.103392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:38.100266623 +0000 UTC m=+462.791320635" watchObservedRunningTime="2026-11-02 17:41:38.10278216 +0000 UTC m=+462.793836172" Nov 02 17:41:38.350958 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:38.350934 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm" Nov 02 17:41:38.373420 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:38.373013 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea9816f4-bad9-4622-a16c-6f7846f72c9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea9816f4-bad9-4622-a16c-6f7846f72c9a-plugins-config-volume\") pod \"ea9816f4-bad9-4622-a16c-6f7846f72c9a\" (UID: \"ea9816f4-bad9-4622-a16c-6f7846f72c9a\") " Nov 02 17:41:38.373420 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:38.373101 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea9816f4-bad9-4622-a16c-6f7846f72c9a-kube-api-access-9mv5r\" (UniqueName: \"kubernetes.io/projected/ea9816f4-bad9-4622-a16c-6f7846f72c9a-kube-api-access-9mv5r\") pod \"ea9816f4-bad9-4622-a16c-6f7846f72c9a\" (UID: \"ea9816f4-bad9-4622-a16c-6f7846f72c9a\") " Nov 02 17:41:38.373629 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:38.373451 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea9816f4-bad9-4622-a16c-6f7846f72c9a-plugins-config-volume" pod "ea9816f4-bad9-4622-a16c-6f7846f72c9a" (UID: "ea9816f4-bad9-4622-a16c-6f7846f72c9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:38.376055 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:38.376022 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9816f4-bad9-4622-a16c-6f7846f72c9a-kube-api-access-9mv5r" pod "ea9816f4-bad9-4622-a16c-6f7846f72c9a" (UID: "ea9816f4-bad9-4622-a16c-6f7846f72c9a"). InnerVolumeSpecName "kube-api-access-9mv5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:38.474781 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:38.474747 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea9816f4-bad9-4622-a16c-6f7846f72c9a-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:38.474781 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:38.474776 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mv5r\" (UniqueName: \"kubernetes.io/projected/ea9816f4-bad9-4622-a16c-6f7846f72c9a-kube-api-access-9mv5r\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:39.063860 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:39.063821 5601 generic.go:356] "Generic (PLEG): container finished" podID="ea9816f4-bad9-4622-a16c-6f7846f72c9a" containerID="f5132738363890611778c6e359e4b6472cbea25197dc1574a2b87f6d7b7962e6" exitCode=0 Nov 02 17:41:39.064376 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:39.063879 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm" Nov 02 17:41:39.064376 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:39.063914 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm" event={"ID":"ea9816f4-bad9-4622-a16c-6f7846f72c9a","Type":"ContainerDied","Data":"f5132738363890611778c6e359e4b6472cbea25197dc1574a2b87f6d7b7962e6"} Nov 02 17:41:39.064376 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:39.063973 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm" event={"ID":"ea9816f4-bad9-4622-a16c-6f7846f72c9a","Type":"ContainerDied","Data":"e9b17e1206c1833d87ef44f094bf6ad6e908d7cc6f2b6467ab04a26e42e74bbe"} Nov 02 17:41:39.064376 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:39.063997 5601 scope.go:122] "RemoveContainer" containerID="f5132738363890611778c6e359e4b6472cbea25197dc1574a2b87f6d7b7962e6" Nov 02 17:41:39.074451 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:39.074434 5601 scope.go:122] "RemoveContainer" containerID="f5132738363890611778c6e359e4b6472cbea25197dc1574a2b87f6d7b7962e6" Nov 02 17:41:39.074794 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:41:39.074762 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5132738363890611778c6e359e4b6472cbea25197dc1574a2b87f6d7b7962e6\": container with ID starting with f5132738363890611778c6e359e4b6472cbea25197dc1574a2b87f6d7b7962e6 not found: ID does not exist" containerID="f5132738363890611778c6e359e4b6472cbea25197dc1574a2b87f6d7b7962e6" Nov 02 17:41:39.074832 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:39.074800 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5132738363890611778c6e359e4b6472cbea25197dc1574a2b87f6d7b7962e6"} err="failed to get container status \"f5132738363890611778c6e359e4b6472cbea25197dc1574a2b87f6d7b7962e6\": rpc error: code = NotFound desc = could not find container \"f5132738363890611778c6e359e4b6472cbea25197dc1574a2b87f6d7b7962e6\": container with ID starting with f5132738363890611778c6e359e4b6472cbea25197dc1574a2b87f6d7b7962e6 not found: ID does not exist" Nov 02 17:41:39.088917 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:39.088892 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm"] Nov 02 17:41:39.097195 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:39.097161 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad831efe-5fbb9fcdd9-gwwrm"] Nov 02 17:41:39.428574 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:39.426691 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea9816f4-bad9-4622-a16c-6f7846f72c9a" path="/var/lib/kubelet/pods/ea9816f4-bad9-4622-a16c-6f7846f72c9a/volumes" Nov 02 17:41:40.975463 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:40.975431 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9"] Nov 02 17:41:40.976048 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:40.976026 5601 cpu_manager.go:414] "containerMap: removing container" podUID="ea9816f4-bad9-4622-a16c-6f7846f72c9a" containerName="payload-processing" Nov 02 17:41:40.976149 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:40.976050 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea9816f4-bad9-4622-a16c-6f7846f72c9a" containerName="payload-processing" Nov 02 17:41:40.976211 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:40.976176 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea9816f4-bad9-4622-a16c-6f7846f72c9a" containerName="payload-processing" Nov 02 17:41:40.981748 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:40.981723 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" Nov 02 17:41:40.989835 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:40.989806 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9"] Nov 02 17:41:41.069346 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:41.069315 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-4b97c6-dockercfg-gjb7h\"" Nov 02 17:41:41.069346 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:41.069315 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-4b97c6\"" Nov 02 17:41:41.100711 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:41.100679 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6fn8\" (UniqueName: \"kubernetes.io/projected/adb37323-1f64-44d2-906c-853342775292-kube-api-access-r6fn8\") pod \"payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9\" (UID: \"adb37323-1f64-44d2-906c-853342775292\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" Nov 02 17:41:41.100875 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:41.100854 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb37323-1f64-44d2-906c-853342775292-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9\" (UID: \"adb37323-1f64-44d2-906c-853342775292\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" Nov 02 17:41:41.202106 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:41.202069 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6fn8\" (UniqueName: \"kubernetes.io/projected/adb37323-1f64-44d2-906c-853342775292-kube-api-access-r6fn8\") pod \"payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9\" (UID: \"adb37323-1f64-44d2-906c-853342775292\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" Nov 02 17:41:41.202261 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:41.202178 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb37323-1f64-44d2-906c-853342775292-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9\" (UID: \"adb37323-1f64-44d2-906c-853342775292\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" Nov 02 17:41:41.202853 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:41.202836 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb37323-1f64-44d2-906c-853342775292-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9\" (UID: \"adb37323-1f64-44d2-906c-853342775292\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" Nov 02 17:41:41.210289 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:41.210260 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6fn8\" (UniqueName: \"kubernetes.io/projected/adb37323-1f64-44d2-906c-853342775292-kube-api-access-r6fn8\") pod \"payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9\" (UID: \"adb37323-1f64-44d2-906c-853342775292\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" Nov 02 17:41:41.299238 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:41.299159 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" Nov 02 17:41:41.583205 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:41.583178 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9"] Nov 02 17:41:41.585028 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:41:41.584992 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb37323_1f64_44d2_906c_853342775292.slice/crio-b6e0078e88d79e680489f784e427d79b3d742d112e781eeb0ceb87fd407e6d64 WatchSource:0}: Error finding container b6e0078e88d79e680489f784e427d79b3d742d112e781eeb0ceb87fd407e6d64: Status 404 returned error can't find the container with id b6e0078e88d79e680489f784e427d79b3d742d112e781eeb0ceb87fd407e6d64 Nov 02 17:41:42.087913 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.087872 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" event={"ID":"adb37323-1f64-44d2-906c-853342775292","Type":"ContainerStarted","Data":"01222030ce3bf1c9c2b872752405066005ffbee16ff02683e7a48bc02087cc73"} Nov 02 17:41:42.088437 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.087943 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" event={"ID":"adb37323-1f64-44d2-906c-853342775292","Type":"ContainerStarted","Data":"b6e0078e88d79e680489f784e427d79b3d742d112e781eeb0ceb87fd407e6d64"} Nov 02 17:41:42.088437 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.088082 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" Nov 02 17:41:42.125796 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.125747 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" podStartSLOduration=2.125731027 podStartE2EDuration="2.125731027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:42.111403954 +0000 UTC m=+466.802457968" watchObservedRunningTime="2026-11-02 17:41:42.125731027 +0000 UTC m=+466.816785039" Nov 02 17:41:42.126973 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.126955 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7"] Nov 02 17:41:42.130897 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.130873 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" Nov 02 17:41:42.140558 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.140237 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-4b97c6-serving-cert\"" Nov 02 17:41:42.143942 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.143915 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7"] Nov 02 17:41:42.214566 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.214519 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxbwt\" (UniqueName: \"kubernetes.io/projected/c2852736-70fa-4810-9f5a-6211fae2d2c9-kube-api-access-jxbwt\") pod \"maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7\" (UID: \"c2852736-70fa-4810-9f5a-6211fae2d2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" Nov 02 17:41:42.214741 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.214704 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2852736-70fa-4810-9f5a-6211fae2d2c9-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7\" (UID: \"c2852736-70fa-4810-9f5a-6211fae2d2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" Nov 02 17:41:42.214900 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.214880 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2852736-70fa-4810-9f5a-6211fae2d2c9-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7\" (UID: \"c2852736-70fa-4810-9f5a-6211fae2d2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" Nov 02 17:41:42.214958 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.214912 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2852736-70fa-4810-9f5a-6211fae2d2c9-metrics-certs\") pod \"maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7\" (UID: \"c2852736-70fa-4810-9f5a-6211fae2d2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" Nov 02 17:41:42.316409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.316370 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxbwt\" (UniqueName: \"kubernetes.io/projected/c2852736-70fa-4810-9f5a-6211fae2d2c9-kube-api-access-jxbwt\") pod \"maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7\" (UID: \"c2852736-70fa-4810-9f5a-6211fae2d2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" Nov 02 17:41:42.316610 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.316550 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2852736-70fa-4810-9f5a-6211fae2d2c9-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7\" (UID: \"c2852736-70fa-4810-9f5a-6211fae2d2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" Nov 02 17:41:42.316728 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:41:42.316684 5601 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-serving-cert: secret "maas-api-e2e-worker-w0-4b97c6-serving-cert" not found Nov 02 17:41:42.316728 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.316683 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2852736-70fa-4810-9f5a-6211fae2d2c9-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7\" (UID: \"c2852736-70fa-4810-9f5a-6211fae2d2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" Nov 02 17:41:42.316728 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.316718 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2852736-70fa-4810-9f5a-6211fae2d2c9-metrics-certs\") pod \"maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7\" (UID: \"c2852736-70fa-4810-9f5a-6211fae2d2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" Nov 02 17:41:42.316875 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:41:42.316736 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2852736-70fa-4810-9f5a-6211fae2d2c9-maas-api-tls podName:c2852736-70fa-4810-9f5a-6211fae2d2c9 nodeName:}" failed. No retries permitted until 2026-11-02 17:41:42.816720976 +0000 UTC m=+467.507774967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2852736-70fa-4810-9f5a-6211fae2d2c9-maas-api-tls") pod "maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" (UID: "c2852736-70fa-4810-9f5a-6211fae2d2c9") : secret "maas-api-e2e-worker-w0-4b97c6-serving-cert" not found Nov 02 17:41:42.317301 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.317280 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2852736-70fa-4810-9f5a-6211fae2d2c9-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7\" (UID: \"c2852736-70fa-4810-9f5a-6211fae2d2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" Nov 02 17:41:42.319069 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.319046 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2852736-70fa-4810-9f5a-6211fae2d2c9-metrics-certs\") pod \"maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7\" (UID: \"c2852736-70fa-4810-9f5a-6211fae2d2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" Nov 02 17:41:42.327382 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.327362 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxbwt\" (UniqueName: \"kubernetes.io/projected/c2852736-70fa-4810-9f5a-6211fae2d2c9-kube-api-access-jxbwt\") pod \"maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7\" (UID: \"c2852736-70fa-4810-9f5a-6211fae2d2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" Nov 02 17:41:42.822083 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.822042 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2852736-70fa-4810-9f5a-6211fae2d2c9-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7\" (UID: \"c2852736-70fa-4810-9f5a-6211fae2d2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" Nov 02 17:41:42.824590 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:42.824563 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2852736-70fa-4810-9f5a-6211fae2d2c9-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7\" (UID: \"c2852736-70fa-4810-9f5a-6211fae2d2c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" Nov 02 17:41:43.046831 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:43.046798 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" Nov 02 17:41:43.321650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:43.321606 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7"] Nov 02 17:41:43.323894 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:41:43.323858 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2852736_70fa_4810_9f5a_6211fae2d2c9.slice/crio-4b5f6b890406e42d6218e44b9ee49b300b8ec63b2147e548872e3b8f86b5ae9b WatchSource:0}: Error finding container 4b5f6b890406e42d6218e44b9ee49b300b8ec63b2147e548872e3b8f86b5ae9b: Status 404 returned error can't find the container with id 4b5f6b890406e42d6218e44b9ee49b300b8ec63b2147e548872e3b8f86b5ae9b Nov 02 17:41:44.103985 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.103892 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" event={"ID":"c2852736-70fa-4810-9f5a-6211fae2d2c9","Type":"ContainerStarted","Data":"28abdcb758a94259b7dd8b55ed54ba2bb71ee3b09b67747c9b33e10ee19b3a54"} Nov 02 17:41:44.103985 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.103946 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" event={"ID":"c2852736-70fa-4810-9f5a-6211fae2d2c9","Type":"ContainerStarted","Data":"4b5f6b890406e42d6218e44b9ee49b300b8ec63b2147e548872e3b8f86b5ae9b"} Nov 02 17:41:44.104207 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.104047 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" Nov 02 17:41:44.123439 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.123389 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" podStartSLOduration=1.702557308 podStartE2EDuration="2.123378417s" totalImagesPullingTime="420.821109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:44.12172131 +0000 UTC m=+468.812775334" watchObservedRunningTime="2026-11-02 17:41:44.123378417 +0000 UTC m=+468.814432429" Nov 02 17:41:44.530345 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.530309 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt"] Nov 02 17:41:44.564055 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.564016 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" Nov 02 17:41:44.565827 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.565793 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt"] Nov 02 17:41:44.569087 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.569060 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-77f103-serving-cert\"" Nov 02 17:41:44.665233 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.665197 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-metrics-certs\") pod \"maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt\" (UID: \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" Nov 02 17:41:44.665443 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.665253 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt\" (UID: \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" Nov 02 17:41:44.665514 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.665449 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9mn8\" (UniqueName: \"kubernetes.io/projected/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-kube-api-access-s9mn8\") pod \"maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt\" (UID: \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" Nov 02 17:41:44.665567 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.665534 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt\" (UID: \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" Nov 02 17:41:44.766752 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.766714 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt\" (UID: \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" Nov 02 17:41:44.766752 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.766763 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-metrics-certs\") pod \"maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt\" (UID: \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" Nov 02 17:41:44.766995 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.766801 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt\" (UID: \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" Nov 02 17:41:44.766995 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.766860 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9mn8\" (UniqueName: \"kubernetes.io/projected/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-kube-api-access-s9mn8\") pod \"maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt\" (UID: \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" Nov 02 17:41:44.767571 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.767545 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt\" (UID: \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" Nov 02 17:41:44.769302 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.769275 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-metrics-certs\") pod \"maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt\" (UID: \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" Nov 02 17:41:44.769407 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.769344 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt\" (UID: \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" Nov 02 17:41:44.778745 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.778723 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9mn8\" (UniqueName: \"kubernetes.io/projected/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-kube-api-access-s9mn8\") pod \"maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt\" (UID: \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" Nov 02 17:41:44.888696 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:44.888408 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" Nov 02 17:41:45.104029 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:45.102463 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" Nov 02 17:41:45.218899 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:45.218855 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt"] Nov 02 17:41:46.116750 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:46.116662 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" event={"ID":"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7","Type":"ContainerStarted","Data":"f7ede08cda3f6cf87edc881129ff8f2e5c5e2a2ca0ae676ebfd7b4ff4bc0d862"} Nov 02 17:41:46.116750 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:46.116716 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" event={"ID":"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7","Type":"ContainerStarted","Data":"ec6b89ef98a17dbc2ef6b5147f49362d728d7e7ea08bc832c9187630d21f8193"} Nov 02 17:41:46.117417 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:46.117397 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" Nov 02 17:41:48.071888 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:48.071860 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" Nov 02 17:41:48.096373 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:48.096310 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" podStartSLOduration=3.630805403 podStartE2EDuration="4.096289177s" totalImagesPullingTime="465.483774ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:46.139239178 +0000 UTC m=+470.830293191" watchObservedRunningTime="2026-11-02 17:41:48.096289177 +0000 UTC m=+472.787343189" Nov 02 17:41:50.117189 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:50.117104 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" Nov 02 17:41:53.092855 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:53.092810 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:41:53.093365 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:53.092910 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" podUID="f26c04ee-0060-4863-b50f-2eed99aad8b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:41:53.100083 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:53.100050 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" Nov 02 17:41:53.138960 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:53.138923 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" Nov 02 17:41:53.857781 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:53.857749 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": read tcp 10.134.0.2:55970->10.134.0.51:15021: read: connection reset by peer" start-of-body= Nov 02 17:41:53.857978 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:53.857854 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" podUID="db8e57a9-62e6-4229-921d-f5a40cde2a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": read tcp 10.134.0.2:55970->10.134.0.51:15021: read: connection reset by peer" Nov 02 17:41:54.197218 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:54.197178 5601 generic.go:356] "Generic (PLEG): container finished" podID="f26c04ee-0060-4863-b50f-2eed99aad8b1" containerID="857d53115e96c7bc173b8372ff4b8b85ab10357ee05cb82a85f0c29bc5b55d53" exitCode=0 Nov 02 17:41:54.197702 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:54.197270 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" event={"ID":"f26c04ee-0060-4863-b50f-2eed99aad8b1","Type":"ContainerDied","Data":"857d53115e96c7bc173b8372ff4b8b85ab10357ee05cb82a85f0c29bc5b55d53"} Nov 02 17:41:54.197813 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:54.197796 5601 scope.go:122] "RemoveContainer" containerID="857d53115e96c7bc173b8372ff4b8b85ab10357ee05cb82a85f0c29bc5b55d53" Nov 02 17:41:54.199238 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:54.199200 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:54.199337 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:54.199294 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:54.199389 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:54.199341 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:54.199424 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:54.199389 5601 generic.go:356] "Generic (PLEG): container finished" podID="db8e57a9-62e6-4229-921d-f5a40cde2a7a" containerID="e3816cb714ea3f3f23240adae24bafd276db6a0a51b6df93dd5432bf1b7d1981" exitCode=0 Nov 02 17:41:54.199465 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:54.199449 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" event={"ID":"db8e57a9-62e6-4229-921d-f5a40cde2a7a","Type":"ContainerDied","Data":"e3816cb714ea3f3f23240adae24bafd276db6a0a51b6df93dd5432bf1b7d1981"} Nov 02 17:41:54.199854 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:54.199839 5601 scope.go:122] "RemoveContainer" containerID="e3816cb714ea3f3f23240adae24bafd276db6a0a51b6df93dd5432bf1b7d1981" Nov 02 17:41:54.200982 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:54.200953 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:54.201052 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:54.201020 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:54.201102 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:54.201053 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:41:55.207716 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:55.207681 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" event={"ID":"f26c04ee-0060-4863-b50f-2eed99aad8b1","Type":"ContainerStarted","Data":"f45cbd85089b41a367d6877f7ba0ecb7890b6e565ff12d24e2e1b1aa700e593f"} Nov 02 17:41:55.209949 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:55.209923 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" event={"ID":"db8e57a9-62e6-4229-921d-f5a40cde2a7a","Type":"ContainerStarted","Data":"da170cbd1d20a185d4a421951af721c985ed5b9830b26e53b9d39c9795f15b82"} Nov 02 17:41:55.440642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:55.439991 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:41:55.440642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:55.440122 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 17:41:55.440642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:55.440174 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" podUID="f26c04ee-0060-4863-b50f-2eed99aad8b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 17:41:55.731259 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:55.731229 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:41:55.731435 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:55.731420 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:41:55.731486 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:55.731458 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" podUID="db8e57a9-62e6-4229-921d-f5a40cde2a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 17:41:56.438640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:56.438023 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 17:41:56.438640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:56.438089 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" podUID="f26c04ee-0060-4863-b50f-2eed99aad8b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 17:41:56.731155 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:56.731069 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:41:56.731155 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:56.731121 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" podUID="db8e57a9-62e6-4229-921d-f5a40cde2a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 17:41:57.462641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:57.462050 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 17:41:57.462641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:57.462119 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" podUID="f26c04ee-0060-4863-b50f-2eed99aad8b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 17:41:57.731766 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:57.731673 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:41:57.731766 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:57.731733 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" podUID="db8e57a9-62e6-4229-921d-f5a40cde2a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 17:41:58.447051 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:58.447012 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 17:41:58.447730 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:58.447696 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" podUID="f26c04ee-0060-4863-b50f-2eed99aad8b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 17:41:58.731504 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:58.731422 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:41:58.732081 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:58.731486 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" podUID="db8e57a9-62e6-4229-921d-f5a40cde2a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 17:41:59.442641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:59.437964 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 17:41:59.442641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:59.438024 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" podUID="f26c04ee-0060-4863-b50f-2eed99aad8b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 17:41:59.731306 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:59.731225 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:41:59.731306 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:41:59.731275 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" podUID="db8e57a9-62e6-4229-921d-f5a40cde2a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 17:42:00.438070 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:00.438029 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 17:42:00.448562 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:00.438683 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" podUID="f26c04ee-0060-4863-b50f-2eed99aad8b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 17:42:00.773914 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:00.773821 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:42:00.774277 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:00.774256 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:42:00.775452 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:00.775422 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 17:42:01.121665 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.121570 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5"] Nov 02 17:42:01.125435 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.125412 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.144688 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.144413 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b97c6\"/\"simulator-w0-4b97c6-kserve-self-signed-certs\"" Nov 02 17:42:01.144688 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.144516 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b97c6\"/\"default-dockercfg-dct95\"" Nov 02 17:42:01.145309 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.145280 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4b97c6\"/\"openshift-service-ca.crt\"" Nov 02 17:42:01.145737 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.145190 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4b97c6\"/\"kube-root-ca.crt\"" Nov 02 17:42:01.148080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.148054 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5"] Nov 02 17:42:01.150140 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.150111 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-dshm\") pod \"simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.150261 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.150183 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-tmp-dir\") pod \"simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.150327 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.150245 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phflk\" (UniqueName: \"kubernetes.io/projected/bde15228-bd90-4a8a-a877-cac31762b285-kube-api-access-phflk\") pod \"simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.150327 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.150307 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bde15228-bd90-4a8a-a877-cac31762b285-tls-certs\") pod \"simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.150449 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.150362 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-home\") pod \"simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.150449 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.150388 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-model-cache\") pod \"simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.251921 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.251874 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-dshm\") pod \"simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.252104 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.251945 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-tmp-dir\") pod \"simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.252104 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.251999 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phflk\" (UniqueName: \"kubernetes.io/projected/bde15228-bd90-4a8a-a877-cac31762b285-kube-api-access-phflk\") pod \"simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.252104 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.252040 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bde15228-bd90-4a8a-a877-cac31762b285-tls-certs\") pod \"simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.252104 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.252087 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-home\") pod \"simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.252244 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.252112 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-model-cache\") pod \"simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.252499 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.252462 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-model-cache\") pod \"simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.252608 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.252591 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-tmp-dir\") pod \"simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.252786 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.252762 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-home\") pod \"simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.255164 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.255136 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-dshm\") pod \"simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.255433 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.255409 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bde15228-bd90-4a8a-a877-cac31762b285-tls-certs\") pod \"simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.262060 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.262028 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phflk\" (UniqueName: \"kubernetes.io/projected/bde15228-bd90-4a8a-a877-cac31762b285-kube-api-access-phflk\") pod \"simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.438349 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.438311 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:01.438822 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.438800 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:42:01.439087 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.439062 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:42:01.439723 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.439704 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 17:42:01.719390 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:01.719360 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5"] Nov 02 17:42:01.722138 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:42:01.722108 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbde15228_bd90_4a8a_a877_cac31762b285.slice/crio-24042b8ec8d15b2071e572a1118c18ec9550f93de8abd11535b5f37384c7baa2 WatchSource:0}: Error finding container 24042b8ec8d15b2071e572a1118c18ec9550f93de8abd11535b5f37384c7baa2: Status 404 returned error can't find the container with id 24042b8ec8d15b2071e572a1118c18ec9550f93de8abd11535b5f37384c7baa2 Nov 02 17:42:02.285081 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:02.284447 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" event={"ID":"bde15228-bd90-4a8a-a877-cac31762b285","Type":"ContainerStarted","Data":"24042b8ec8d15b2071e572a1118c18ec9550f93de8abd11535b5f37384c7baa2"} Nov 02 17:42:03.298454 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:03.298356 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" event={"ID":"bde15228-bd90-4a8a-a877-cac31762b285","Type":"ContainerStarted","Data":"38dd9bf500176a0efcfbc8e932252a6c050dd474ad7516ba3aa5f0dbb21af213"} Nov 02 17:42:03.318853 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:03.318800 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podStartSLOduration=1.117251383 podStartE2EDuration="2.318787305s" totalImagesPullingTime="1.201535922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:03.31703098 +0000 UTC m=+488.008084996" watchObservedRunningTime="2026-11-02 17:42:03.318787305 +0000 UTC m=+488.009841317" Nov 02 17:42:03.439445 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:03.439403 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:04.483604 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:42:04.483569 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:42:06.607007 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.606973 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc"] Nov 02 17:42:06.610320 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.610299 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:06.617669 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.617641 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-24cfb6\"/\"openshift-service-ca.crt\"" Nov 02 17:42:06.617834 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.617815 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-24cfb6\"/\"simulator-w1-24cfb6-kserve-self-signed-certs\"" Nov 02 17:42:06.618362 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.617948 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-24cfb6\"/\"kube-root-ca.crt\"" Nov 02 17:42:06.618362 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.618024 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-24cfb6\"/\"default-dockercfg-gdgnd\"" Nov 02 17:42:06.620529 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.620506 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc"] Nov 02 17:42:06.710373 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.710333 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-home\") pod \"simulator-w1-24cfb6-kserve-695c745d89-q4nfc\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:06.710553 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.710389 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436816c2-2c00-4cad-aac7-298db2f5e8d5-tls-certs\") pod \"simulator-w1-24cfb6-kserve-695c745d89-q4nfc\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:06.710553 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.710481 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjzt5\" (UniqueName: \"kubernetes.io/projected/436816c2-2c00-4cad-aac7-298db2f5e8d5-kube-api-access-cjzt5\") pod \"simulator-w1-24cfb6-kserve-695c745d89-q4nfc\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:06.710667 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.710645 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-tmp-dir\") pod \"simulator-w1-24cfb6-kserve-695c745d89-q4nfc\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:06.710724 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.710694 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-dshm\") pod \"simulator-w1-24cfb6-kserve-695c745d89-q4nfc\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:06.710771 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.710761 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-model-cache\") pod \"simulator-w1-24cfb6-kserve-695c745d89-q4nfc\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:06.812105 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.812056 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-dshm\") pod \"simulator-w1-24cfb6-kserve-695c745d89-q4nfc\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:06.812312 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.812118 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-model-cache\") pod \"simulator-w1-24cfb6-kserve-695c745d89-q4nfc\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:06.812312 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.812203 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-home\") pod \"simulator-w1-24cfb6-kserve-695c745d89-q4nfc\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:06.812312 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.812244 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436816c2-2c00-4cad-aac7-298db2f5e8d5-tls-certs\") pod \"simulator-w1-24cfb6-kserve-695c745d89-q4nfc\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:06.812312 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.812269 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjzt5\" (UniqueName: \"kubernetes.io/projected/436816c2-2c00-4cad-aac7-298db2f5e8d5-kube-api-access-cjzt5\") pod \"simulator-w1-24cfb6-kserve-695c745d89-q4nfc\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:06.812539 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.812347 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-tmp-dir\") pod \"simulator-w1-24cfb6-kserve-695c745d89-q4nfc\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:06.812672 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.812603 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-model-cache\") pod \"simulator-w1-24cfb6-kserve-695c745d89-q4nfc\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:06.812672 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.812611 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-home\") pod \"simulator-w1-24cfb6-kserve-695c745d89-q4nfc\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:06.812797 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.812714 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-tmp-dir\") pod \"simulator-w1-24cfb6-kserve-695c745d89-q4nfc\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:06.814379 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.814358 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-dshm\") pod \"simulator-w1-24cfb6-kserve-695c745d89-q4nfc\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:06.814638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.814601 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436816c2-2c00-4cad-aac7-298db2f5e8d5-tls-certs\") pod \"simulator-w1-24cfb6-kserve-695c745d89-q4nfc\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:06.821313 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.821289 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjzt5\" (UniqueName: \"kubernetes.io/projected/436816c2-2c00-4cad-aac7-298db2f5e8d5-kube-api-access-cjzt5\") pod \"simulator-w1-24cfb6-kserve-695c745d89-q4nfc\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:06.925223 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:06.925190 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:07.200395 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:07.200369 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc"] Nov 02 17:42:07.202955 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:42:07.202912 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod436816c2_2c00_4cad_aac7_298db2f5e8d5.slice/crio-335a70147f49963416111982d947dea6b6157741f0bcc25730d29f7cc532ead3 WatchSource:0}: Error finding container 335a70147f49963416111982d947dea6b6157741f0bcc25730d29f7cc532ead3: Status 404 returned error can't find the container with id 335a70147f49963416111982d947dea6b6157741f0bcc25730d29f7cc532ead3 Nov 02 17:42:07.324468 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:07.324423 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" event={"ID":"436816c2-2c00-4cad-aac7-298db2f5e8d5","Type":"ContainerStarted","Data":"26e80b94447307bbb7d9c68eff1e3adc0eaf57963b87b71a0dea35da0f08abf2"} Nov 02 17:42:07.324670 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:07.324478 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" event={"ID":"436816c2-2c00-4cad-aac7-298db2f5e8d5","Type":"ContainerStarted","Data":"335a70147f49963416111982d947dea6b6157741f0bcc25730d29f7cc532ead3"} Nov 02 17:42:07.343158 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:07.343093 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podStartSLOduration=1.343075563 podStartE2EDuration="1.343075563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:07.341856387 +0000 UTC m=+492.032910404" watchObservedRunningTime="2026-11-02 17:42:07.343075563 +0000 UTC m=+492.034129577" Nov 02 17:42:07.925320 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:07.925286 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:11.439216 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:11.439186 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:11.452818 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:11.452793 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:12.367579 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.367548 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:42:12.686668 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.686595 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6"] Nov 02 17:42:12.690112 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.690088 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:12.698272 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.698031 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a37cf0\"/\"openshift-service-ca.crt\"" Nov 02 17:42:12.700134 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.698594 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a37cf0\"/\"default-dockercfg-k8pxt\"" Nov 02 17:42:12.700134 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.698758 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a37cf0\"/\"premium-w3-a37cf0-kserve-self-signed-certs\"" Nov 02 17:42:12.700435 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.700411 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a37cf0\"/\"kube-root-ca.crt\"" Nov 02 17:42:12.701028 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.701008 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6"] Nov 02 17:42:12.771939 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.771905 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-model-cache\") pod \"premium-w3-a37cf0-kserve-8d64b668c-96tj6\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:12.772114 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.771942 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-tmp-dir\") pod \"premium-w3-a37cf0-kserve-8d64b668c-96tj6\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:12.772114 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.772076 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-dshm\") pod \"premium-w3-a37cf0-kserve-8d64b668c-96tj6\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:12.772241 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.772114 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c749700-fae4-498d-9fb1-38652eec3eef-tls-certs\") pod \"premium-w3-a37cf0-kserve-8d64b668c-96tj6\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:12.772241 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.772217 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-home\") pod \"premium-w3-a37cf0-kserve-8d64b668c-96tj6\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:12.772336 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.772327 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp254\" (UniqueName: \"kubernetes.io/projected/5c749700-fae4-498d-9fb1-38652eec3eef-kube-api-access-hp254\") pod \"premium-w3-a37cf0-kserve-8d64b668c-96tj6\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:12.873598 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.873559 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-dshm\") pod \"premium-w3-a37cf0-kserve-8d64b668c-96tj6\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:12.873818 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.873610 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c749700-fae4-498d-9fb1-38652eec3eef-tls-certs\") pod \"premium-w3-a37cf0-kserve-8d64b668c-96tj6\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:12.873818 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.873670 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-home\") pod \"premium-w3-a37cf0-kserve-8d64b668c-96tj6\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:12.873818 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.873710 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp254\" (UniqueName: \"kubernetes.io/projected/5c749700-fae4-498d-9fb1-38652eec3eef-kube-api-access-hp254\") pod \"premium-w3-a37cf0-kserve-8d64b668c-96tj6\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:12.873818 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.873737 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-model-cache\") pod \"premium-w3-a37cf0-kserve-8d64b668c-96tj6\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:12.873818 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.873752 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-tmp-dir\") pod \"premium-w3-a37cf0-kserve-8d64b668c-96tj6\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:12.874151 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.874127 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-home\") pod \"premium-w3-a37cf0-kserve-8d64b668c-96tj6\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:12.874194 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.874147 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-model-cache\") pod \"premium-w3-a37cf0-kserve-8d64b668c-96tj6\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:12.874232 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.874197 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-tmp-dir\") pod \"premium-w3-a37cf0-kserve-8d64b668c-96tj6\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:12.876034 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.875912 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-dshm\") pod \"premium-w3-a37cf0-kserve-8d64b668c-96tj6\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:12.877165 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.877146 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c749700-fae4-498d-9fb1-38652eec3eef-tls-certs\") pod \"premium-w3-a37cf0-kserve-8d64b668c-96tj6\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:12.882702 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:12.882676 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp254\" (UniqueName: \"kubernetes.io/projected/5c749700-fae4-498d-9fb1-38652eec3eef-kube-api-access-hp254\") pod \"premium-w3-a37cf0-kserve-8d64b668c-96tj6\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:13.004001 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:13.003909 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:13.280240 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:13.280212 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6"] Nov 02 17:42:13.282386 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:42:13.282358 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c749700_fae4_498d_9fb1_38652eec3eef.slice/crio-28647a607b76d0e31c6dea82099c4e0b2270c28e16a1d5ec1601c551e8294dd7 WatchSource:0}: Error finding container 28647a607b76d0e31c6dea82099c4e0b2270c28e16a1d5ec1601c551e8294dd7: Status 404 returned error can't find the container with id 28647a607b76d0e31c6dea82099c4e0b2270c28e16a1d5ec1601c551e8294dd7 Nov 02 17:42:13.361374 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:13.361341 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" event={"ID":"5c749700-fae4-498d-9fb1-38652eec3eef","Type":"ContainerStarted","Data":"ca1c8f2ce9f27230e354ac817f1246aacb41e71b53b4b8efd62ddf21d94d1c93"} Nov 02 17:42:13.361545 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:13.361384 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" event={"ID":"5c749700-fae4-498d-9fb1-38652eec3eef","Type":"ContainerStarted","Data":"28647a607b76d0e31c6dea82099c4e0b2270c28e16a1d5ec1601c551e8294dd7"} Nov 02 17:42:13.381132 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:13.381081 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podStartSLOduration=1.381067367 podStartE2EDuration="1.381067367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:13.378109976 +0000 UTC m=+498.069163990" watchObservedRunningTime="2026-11-02 17:42:13.381067367 +0000 UTC m=+498.072121379" Nov 02 17:42:14.004234 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:14.004186 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:16.926262 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:16.926220 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:16.939584 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:16.939554 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:17.398473 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:17.398393 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:42:23.004926 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:23.004879 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:23.023604 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:23.023574 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:23.439285 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:42:23.439257 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:42:34.483209 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:42:34.483167 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:43:04.483840 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:43:04.483801 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:43:06.742577 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:06.742540 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft"] Nov 02 17:43:06.747579 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:06.747556 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:06.752756 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:06.752734 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-24cfb6-openshift-default-dockercfg-r2v2f\"" Nov 02 17:43:06.765367 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:06.765331 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft"] Nov 02 17:43:06.932014 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:06.931979 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:06.932192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:06.932034 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:06.932192 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:06.932057 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:06.932273 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:06.932200 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgfrf\" (UniqueName: \"kubernetes.io/projected/5c1fcd8f-3977-4d4b-826c-38150e08228c-kube-api-access-mgfrf\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:06.932307 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:06.932277 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:06.932340 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:06.932308 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:06.932373 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:06.932343 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:06.932505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:06.932485 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:06.932544 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:06.932515 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1fcd8f-3977-4d4b-826c-38150e08228c-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.034075 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.033980 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.034075 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.034029 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.034075 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.034059 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgfrf\" (UniqueName: \"kubernetes.io/projected/5c1fcd8f-3977-4d4b-826c-38150e08228c-kube-api-access-mgfrf\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.034355 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.034106 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.034355 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.034277 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.034355 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.034339 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.034516 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.034450 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.034516 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.034483 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1fcd8f-3977-4d4b-826c-38150e08228c-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.034634 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.034531 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.034697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.034635 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.034854 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.034821 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.034938 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.034856 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.034938 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.034907 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.035208 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.035186 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1fcd8f-3977-4d4b-826c-38150e08228c-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.036756 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.036725 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.036847 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.036794 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.043883 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.043862 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.044094 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.044072 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgfrf\" (UniqueName: \"kubernetes.io/projected/5c1fcd8f-3977-4d4b-826c-38150e08228c-kube-api-access-mgfrf\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.060750 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.060723 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:07.345151 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.345124 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft"] Nov 02 17:43:07.352384 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.352341 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:07.352491 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.352440 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:07.352491 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.352479 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:07.681032 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.680995 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" event={"ID":"5c1fcd8f-3977-4d4b-826c-38150e08228c","Type":"ContainerStarted","Data":"4dc8ac7a3054619e9036846edbf077daab0d8aa87061a7552e1ba57ff52abeb1"} Nov 02 17:43:07.681217 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.681046 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" event={"ID":"5c1fcd8f-3977-4d4b-826c-38150e08228c","Type":"ContainerStarted","Data":"18b78f655ff3f4273b9927441b8dd81567c1983785a637b4bb039c419fa6559a"} Nov 02 17:43:07.709767 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:07.709703 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" podStartSLOduration=1.7096831350000001 podStartE2EDuration="1.709683135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:07.707368692 +0000 UTC m=+552.398422707" watchObservedRunningTime="2026-11-02 17:43:07.709683135 +0000 UTC m=+552.400737148" Nov 02 17:43:08.064792 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:08.064704 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:08.066333 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:08.066307 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:43:08.066557 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:08.066522 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" podUID="5c1fcd8f-3977-4d4b-826c-38150e08228c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 17:43:09.062060 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:09.062018 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:43:09.062261 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:09.062084 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" podUID="5c1fcd8f-3977-4d4b-826c-38150e08228c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 17:43:10.063642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:10.062200 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:43:10.063642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:10.062265 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" podUID="5c1fcd8f-3977-4d4b-826c-38150e08228c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 17:43:11.061418 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:11.061377 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:43:11.061648 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:11.061456 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" podUID="5c1fcd8f-3977-4d4b-826c-38150e08228c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 17:43:12.062011 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:12.061973 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:43:12.062442 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:12.062033 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" podUID="5c1fcd8f-3977-4d4b-826c-38150e08228c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 17:43:13.061470 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:13.061438 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:43:13.061666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:13.061495 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" podUID="5c1fcd8f-3977-4d4b-826c-38150e08228c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 17:43:14.071179 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:14.071142 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:14.071825 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:14.071569 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:14.072672 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:14.072645 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:43:30.540107 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.540070 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7"] Nov 02 17:43:30.544381 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.544359 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.555186 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.555154 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a37cf0-openshift-default-dockercfg-mbxtp\"" Nov 02 17:43:30.557910 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.557878 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7"] Nov 02 17:43:30.628610 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.628571 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-podinfo\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.628610 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.628608 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istiod-ca-cert\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.628836 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.628650 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-workload-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.628872 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.628835 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-envoy\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.628906 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.628869 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-data\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.628944 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.628901 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjfvz\" (UniqueName: \"kubernetes.io/projected/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-kube-api-access-gjfvz\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.628985 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.628968 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-token\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.629037 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.629023 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-credential-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.629072 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.629064 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-workload-certs\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.730372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.730329 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjfvz\" (UniqueName: \"kubernetes.io/projected/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-kube-api-access-gjfvz\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.730582 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.730388 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-token\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.730582 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.730435 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-credential-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.730582 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.730457 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-workload-certs\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.730582 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.730483 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-podinfo\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.730582 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.730498 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istiod-ca-cert\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.730582 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.730526 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-workload-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.730957 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.730655 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-envoy\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.730957 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.730684 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-data\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.731066 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.730991 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-credential-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.731202 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.731175 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-workload-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.731368 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.731342 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istiod-ca-cert\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.731368 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.731194 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-data\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.731531 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.731244 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-workload-certs\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.732946 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.732921 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-envoy\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.733180 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.733163 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-podinfo\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.739042 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.739018 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-token\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.739229 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.739209 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjfvz\" (UniqueName: \"kubernetes.io/projected/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-kube-api-access-gjfvz\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:30.861938 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:30.861853 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:31.149496 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.149461 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7"] Nov 02 17:43:31.151723 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:43:31.151684 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ac5b40_0dae_41ea_a7ca_c7c2313d692f.slice/crio-bb248fdaf3167c756da4d3c2e9315a68c0566ecdb26e1bb9666aaaa5ecf5a104 WatchSource:0}: Error finding container bb248fdaf3167c756da4d3c2e9315a68c0566ecdb26e1bb9666aaaa5ecf5a104: Status 404 returned error can't find the container with id bb248fdaf3167c756da4d3c2e9315a68c0566ecdb26e1bb9666aaaa5ecf5a104 Nov 02 17:43:31.154468 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.154431 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:31.154577 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.154509 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:31.154577 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.154560 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:31.219207 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.219176 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps"] Nov 02 17:43:31.223707 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.223685 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.246943 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.246911 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps"] Nov 02 17:43:31.247535 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.247513 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-24cfb6\"/\"e2e-conflict-a-f1a82c40-kserve-self-signed-certs\"" Nov 02 17:43:31.339643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.339593 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-model-cache\") pod \"e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.339826 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.339702 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81707204-a73d-4ca8-bdc1-72995f633c36-tls-certs\") pod \"e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.339826 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.339729 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-home\") pod \"e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.339826 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.339770 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-tmp-dir\") pod \"e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.340008 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.339853 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-dshm\") pod \"e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.340008 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.339933 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk8ql\" (UniqueName: \"kubernetes.io/projected/81707204-a73d-4ca8-bdc1-72995f633c36-kube-api-access-bk8ql\") pod \"e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.447640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.440993 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk8ql\" (UniqueName: \"kubernetes.io/projected/81707204-a73d-4ca8-bdc1-72995f633c36-kube-api-access-bk8ql\") pod \"e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.447640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.441093 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-model-cache\") pod \"e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.447640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.441156 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81707204-a73d-4ca8-bdc1-72995f633c36-tls-certs\") pod \"e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.447640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.441249 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-home\") pod \"e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.447640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.441282 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-tmp-dir\") pod \"e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.447640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.441324 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-dshm\") pod \"e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.447640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.441476 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-model-cache\") pod \"e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.447640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.442108 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-home\") pod \"e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.447640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.442935 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-tmp-dir\") pod \"e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.447640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.445162 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81707204-a73d-4ca8-bdc1-72995f633c36-tls-certs\") pod \"e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.451194 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.450199 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk8ql\" (UniqueName: \"kubernetes.io/projected/81707204-a73d-4ca8-bdc1-72995f633c36-kube-api-access-bk8ql\") pod \"e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.451194 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.450500 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-dshm\") pod \"e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.552904 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.552871 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:31.833277 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.833188 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" event={"ID":"42ac5b40-0dae-41ea-a7ca-c7c2313d692f","Type":"ContainerStarted","Data":"18a062f47e0feba21bc4fd6f27807a70bf20f148b5e9e208bc41c1cb99e72687"} Nov 02 17:43:31.833277 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.833252 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" event={"ID":"42ac5b40-0dae-41ea-a7ca-c7c2313d692f","Type":"ContainerStarted","Data":"bb248fdaf3167c756da4d3c2e9315a68c0566ecdb26e1bb9666aaaa5ecf5a104"} Nov 02 17:43:31.869895 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.866530 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps"] Nov 02 17:43:31.869895 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:31.866606 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:32.864669 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:32.864637 5601 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:43:32.865139 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:32.864691 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" podUID="42ac5b40-0dae-41ea-a7ca-c7c2313d692f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 17:43:32.865139 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:32.865031 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" event={"ID":"81707204-a73d-4ca8-bdc1-72995f633c36","Type":"ContainerStarted","Data":"41d03f69322046a23a1f74ab0e42fc353b3e989ef907113b376e15137ec366d5"} Nov 02 17:43:32.865139 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:32.865074 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" event={"ID":"81707204-a73d-4ca8-bdc1-72995f633c36","Type":"ContainerStarted","Data":"c0babf813abdb35095ce34ff5e7aac05ced1c55a776ed75c78bee8f58553e593"} Nov 02 17:43:32.887240 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:32.887191 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" podStartSLOduration=1.887177967 podStartE2EDuration="1.887177967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:32.885810355 +0000 UTC m=+577.576864367" watchObservedRunningTime="2026-11-02 17:43:32.887177967 +0000 UTC m=+577.578231978" Nov 02 17:43:32.887770 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:32.887742 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" podStartSLOduration=2.887735229 podStartE2EDuration="2.887735229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:31.889879962 +0000 UTC m=+576.580933971" watchObservedRunningTime="2026-11-02 17:43:32.887735229 +0000 UTC m=+577.578789240" Nov 02 17:43:33.559782 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:33.559739 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:33.863217 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:33.863133 5601 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:43:33.863217 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:33.863189 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" podUID="42ac5b40-0dae-41ea-a7ca-c7c2313d692f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 17:43:34.487717 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:43:34.487674 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:43:34.863199 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:34.863118 5601 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:43:34.863199 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:34.863184 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" podUID="42ac5b40-0dae-41ea-a7ca-c7c2313d692f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 17:43:35.863287 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:35.863183 5601 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:43:35.863287 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:35.863247 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" podUID="42ac5b40-0dae-41ea-a7ca-c7c2313d692f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 17:43:36.863285 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:36.863240 5601 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:43:36.863738 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:36.863309 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" podUID="42ac5b40-0dae-41ea-a7ca-c7c2313d692f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 17:43:38.032640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:38.031925 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:38.036633 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:38.033323 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:38.040632 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:38.038130 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 17:43:41.553392 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:41.553350 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:41.566261 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:41.566232 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:41.937676 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:41.937642 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:43:50.807066 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:50.807013 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz"] Nov 02 17:43:50.811183 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:50.811159 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:50.822743 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:50.822703 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b97c6\"/\"distinct-w0-4b97c6-kserve-self-signed-certs\"" Nov 02 17:43:50.825185 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:50.825157 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz"] Nov 02 17:43:50.913162 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:50.913126 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76064488-6a4c-4f85-b1ee-75388968d2a1-tls-certs\") pod \"distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:50.913430 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:50.913396 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-home\") pod \"distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:50.913577 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:50.913450 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-dshm\") pod \"distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:50.913577 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:50.913570 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-model-cache\") pod \"distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:50.913733 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:50.913636 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-tmp-dir\") pod \"distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:50.913733 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:50.913699 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrmtl\" (UniqueName: \"kubernetes.io/projected/76064488-6a4c-4f85-b1ee-75388968d2a1-kube-api-access-lrmtl\") pod \"distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:51.014657 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:51.014603 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-model-cache\") pod \"distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:51.014836 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:51.014692 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-tmp-dir\") pod \"distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:51.014836 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:51.014750 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmtl\" (UniqueName: \"kubernetes.io/projected/76064488-6a4c-4f85-b1ee-75388968d2a1-kube-api-access-lrmtl\") pod \"distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:51.014836 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:51.014832 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76064488-6a4c-4f85-b1ee-75388968d2a1-tls-certs\") pod \"distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:51.014979 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:51.014956 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-home\") pod \"distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:51.015039 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:51.015016 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-dshm\") pod \"distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:51.015091 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:51.015042 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-model-cache\") pod \"distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:51.015154 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:51.015080 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-tmp-dir\") pod \"distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:51.015256 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:51.015233 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-home\") pod \"distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:51.017207 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:51.017177 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-dshm\") pod \"distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:51.017505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:51.017483 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76064488-6a4c-4f85-b1ee-75388968d2a1-tls-certs\") pod \"distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:51.023813 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:51.023793 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrmtl\" (UniqueName: \"kubernetes.io/projected/76064488-6a4c-4f85-b1ee-75388968d2a1-kube-api-access-lrmtl\") pod \"distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:51.125001 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:51.124916 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:51.405904 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:51.405877 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz"] Nov 02 17:43:52.018359 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.018325 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" event={"ID":"76064488-6a4c-4f85-b1ee-75388968d2a1","Type":"ContainerStarted","Data":"c2c54e0df990010f5261eee414ae00e9c8cc5a2187de64e70a377a9bc25f5337"} Nov 02 17:43:52.019019 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.018368 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" event={"ID":"76064488-6a4c-4f85-b1ee-75388968d2a1","Type":"ContainerStarted","Data":"4612491f854588242fc420eecf966240283fd4ed678598d45b55bf5b044537f2"} Nov 02 17:43:52.038595 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.038545 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podStartSLOduration=2.038532786 podStartE2EDuration="2.038532786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:52.036398155 +0000 UTC m=+596.727452169" watchObservedRunningTime="2026-11-02 17:43:52.038532786 +0000 UTC m=+596.729586798" Nov 02 17:43:52.125978 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.125944 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:43:52.240284 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.240246 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk"] Nov 02 17:43:52.245206 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.245176 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.263300 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.263268 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk"] Nov 02 17:43:52.331365 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.331275 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.331365 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.331318 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f57ca2f-9916-45aa-9523-65e8fd925f15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.331365 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.331354 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.331588 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.331465 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.331687 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.331669 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.331745 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.331731 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.331801 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.331786 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqdlm\" (UniqueName: \"kubernetes.io/projected/2f57ca2f-9916-45aa-9523-65e8fd925f15-kube-api-access-lqdlm\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.331876 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.331855 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.331910 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.331887 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.433236 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.433186 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.433448 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.433271 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.433448 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.433302 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqdlm\" (UniqueName: \"kubernetes.io/projected/2f57ca2f-9916-45aa-9523-65e8fd925f15-kube-api-access-lqdlm\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.434019 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.433991 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.434156 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.434122 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.435342 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.435302 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.435467 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.435368 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.435538 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.435480 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.435640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.435511 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f57ca2f-9916-45aa-9523-65e8fd925f15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.435708 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.435690 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.435747 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.435726 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.436430 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.436403 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.436755 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.436678 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.436755 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.436706 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.437524 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.437485 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f57ca2f-9916-45aa-9523-65e8fd925f15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.440509 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.440482 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.448587 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.448556 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqdlm\" (UniqueName: \"kubernetes.io/projected/2f57ca2f-9916-45aa-9523-65e8fd925f15-kube-api-access-lqdlm\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.449395 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.449375 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-mdxxk\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.567488 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.567457 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:52.852425 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.852365 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk"] Nov 02 17:43:52.853715 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:43:52.853687 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f57ca2f_9916_45aa_9523_65e8fd925f15.slice/crio-f980fad24c157423ce6af8646a3b5eaebe85e76f9a0dd787111ca5a420d6f7f6 WatchSource:0}: Error finding container f980fad24c157423ce6af8646a3b5eaebe85e76f9a0dd787111ca5a420d6f7f6: Status 404 returned error can't find the container with id f980fad24c157423ce6af8646a3b5eaebe85e76f9a0dd787111ca5a420d6f7f6 Nov 02 17:43:52.856297 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.856257 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:52.856420 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.856334 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:52.856420 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:52.856366 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:43:53.026679 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:53.026535 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" event={"ID":"2f57ca2f-9916-45aa-9523-65e8fd925f15","Type":"ContainerStarted","Data":"616e48855ee28f64839bb98fd48d9e39dd94c7d13b340d72d80b615200b193c8"} Nov 02 17:43:53.026679 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:53.026610 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" event={"ID":"2f57ca2f-9916-45aa-9523-65e8fd925f15","Type":"ContainerStarted","Data":"f980fad24c157423ce6af8646a3b5eaebe85e76f9a0dd787111ca5a420d6f7f6"} Nov 02 17:43:53.053493 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:53.053441 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" podStartSLOduration=1.053424943 podStartE2EDuration="1.053424943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:53.051730128 +0000 UTC m=+597.742784141" watchObservedRunningTime="2026-11-02 17:43:53.053424943 +0000 UTC m=+597.744478955" Nov 02 17:43:53.568155 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:53.568110 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:43:53.569913 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:53.569883 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-mdxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:43:53.570044 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:53.569935 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" podUID="2f57ca2f-9916-45aa-9523-65e8fd925f15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 17:43:54.568212 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:54.568167 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-mdxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:43:54.568677 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:54.568242 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" podUID="2f57ca2f-9916-45aa-9523-65e8fd925f15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 17:43:55.568778 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:55.568749 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-mdxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:43:55.569180 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:55.568799 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" podUID="2f57ca2f-9916-45aa-9523-65e8fd925f15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 17:43:56.568338 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:56.568300 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-mdxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:43:56.568539 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:56.568371 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" podUID="2f57ca2f-9916-45aa-9523-65e8fd925f15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 17:43:57.568857 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:57.568817 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-mdxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:43:57.569235 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:57.568878 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" podUID="2f57ca2f-9916-45aa-9523-65e8fd925f15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 17:43:58.568208 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.568170 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-mdxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:43:58.568424 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.568242 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" podUID="2f57ca2f-9916-45aa-9523-65e8fd925f15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 17:43:58.590811 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.590783 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg"] Nov 02 17:43:58.596653 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.596609 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:58.606832 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.606805 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg"] Nov 02 17:43:58.606966 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.606944 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-24cfb6\"/\"e2e-conflict-b-f1a82c40-kserve-self-signed-certs\"" Nov 02 17:43:58.720199 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.720155 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-model-cache\") pod \"e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:58.720395 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.720214 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b697h\" (UniqueName: \"kubernetes.io/projected/691f60d7-7f1b-4d75-9893-9060513dc151-kube-api-access-b697h\") pod \"e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:58.720395 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.720359 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/691f60d7-7f1b-4d75-9893-9060513dc151-tls-certs\") pod \"e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:58.720395 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.720388 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-dshm\") pod \"e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:58.720547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.720424 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-home\") pod \"e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:58.720547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.720508 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-tmp-dir\") pod \"e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:58.821154 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.821073 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-model-cache\") pod \"e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:58.821154 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.821146 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b697h\" (UniqueName: \"kubernetes.io/projected/691f60d7-7f1b-4d75-9893-9060513dc151-kube-api-access-b697h\") pod \"e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:58.821374 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.821204 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/691f60d7-7f1b-4d75-9893-9060513dc151-tls-certs\") pod \"e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:58.821374 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.821226 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-dshm\") pod \"e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:58.821374 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.821265 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-home\") pod \"e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:58.821374 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.821335 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-tmp-dir\") pod \"e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:58.821631 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.821509 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-model-cache\") pod \"e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:58.821762 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.821732 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-tmp-dir\") pod \"e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:58.821852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.821823 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-home\") pod \"e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:58.823562 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.823533 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-dshm\") pod \"e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:58.823852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.823835 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/691f60d7-7f1b-4d75-9893-9060513dc151-tls-certs\") pod \"e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:58.831565 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.831540 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b697h\" (UniqueName: \"kubernetes.io/projected/691f60d7-7f1b-4d75-9893-9060513dc151-kube-api-access-b697h\") pod \"e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:58.940640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:58.927498 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:43:59.266582 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:59.266547 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg"] Nov 02 17:43:59.267904 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:43:59.267873 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod691f60d7_7f1b_4d75_9893_9060513dc151.slice/crio-6acdd573fed341aa0ff61acba757b39f1dea320ce6c016082589290c64105874 WatchSource:0}: Error finding container 6acdd573fed341aa0ff61acba757b39f1dea320ce6c016082589290c64105874: Status 404 returned error can't find the container with id 6acdd573fed341aa0ff61acba757b39f1dea320ce6c016082589290c64105874 Nov 02 17:43:59.568773 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:59.568679 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-mdxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:43:59.568773 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:43:59.568746 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" podUID="2f57ca2f-9916-45aa-9523-65e8fd925f15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 17:44:00.104025 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:00.103990 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" event={"ID":"691f60d7-7f1b-4d75-9893-9060513dc151","Type":"ContainerStarted","Data":"c061319a8fd6b45f16612f35a6414417ffc94e7caa2e470ecdd18205d7b37136"} Nov 02 17:44:00.104522 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:00.104034 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" event={"ID":"691f60d7-7f1b-4d75-9893-9060513dc151","Type":"ContainerStarted","Data":"6acdd573fed341aa0ff61acba757b39f1dea320ce6c016082589290c64105874"} Nov 02 17:44:00.128623 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:00.128560 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" podStartSLOduration=2.128541641 podStartE2EDuration="2.128541641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:00.125228548 +0000 UTC m=+604.816282560" watchObservedRunningTime="2026-11-02 17:44:00.128541641 +0000 UTC m=+604.819595654" Nov 02 17:44:00.568670 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:00.568606 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-mdxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 17:44:00.568869 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:00.568708 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" podUID="2f57ca2f-9916-45aa-9523-65e8fd925f15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 17:44:00.928200 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:00.928164 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:44:01.125230 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:01.125197 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:44:01.138158 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:01.138122 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:44:01.711925 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:01.711890 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:44:01.712609 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:01.712587 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:44:01.713647 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:01.713599 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 17:44:02.126751 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:02.126664 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:44:04.485582 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:44:04.485534 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:44:08.933029 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:08.932982 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:44:09.255078 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.254990 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:44:09.388641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.384730 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:44:09.757992 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.757957 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl"] Nov 02 17:44:09.762573 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.762531 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:09.770236 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.769608 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b97c6\"/\"distinct-2-w0-4b97c6-kserve-self-signed-certs\"" Nov 02 17:44:09.821790 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.821749 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl"] Nov 02 17:44:09.851642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.841008 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-home\") pod \"distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:09.851949 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.851921 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-model-cache\") pod \"distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:09.856971 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.852155 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-tmp-dir\") pod \"distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:09.857246 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.857220 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-tls-certs\") pod \"distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:09.858821 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.857368 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-dshm\") pod \"distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:09.861813 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.861785 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tjc7\" (UniqueName: \"kubernetes.io/projected/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-kube-api-access-6tjc7\") pod \"distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:09.967642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.967026 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tjc7\" (UniqueName: \"kubernetes.io/projected/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-kube-api-access-6tjc7\") pod \"distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:09.967642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.967170 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-home\") pod \"distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:09.967642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.967224 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-model-cache\") pod \"distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:09.967642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.967275 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-tmp-dir\") pod \"distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:09.967642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.967312 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-tls-certs\") pod \"distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:09.967642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.967336 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-dshm\") pod \"distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:09.968654 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.968486 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-tmp-dir\") pod \"distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:09.969151 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.968771 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-home\") pod \"distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:09.975631 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:09.969549 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-model-cache\") pod \"distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:10.000841 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:10.000805 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-dshm\") pod \"distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:10.005338 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:10.005269 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-tls-certs\") pod \"distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:10.012309 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:10.012195 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tjc7\" (UniqueName: \"kubernetes.io/projected/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-kube-api-access-6tjc7\") pod \"distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:10.147398 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:10.147359 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:10.921003 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:10.918241 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl"] Nov 02 17:44:11.328013 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:11.327926 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" event={"ID":"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff","Type":"ContainerStarted","Data":"07d61aadac87bf905d27d2818bc0cc1e8a1e92ed2bff73a00e77fe4729843537"} Nov 02 17:44:11.328542 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:11.328497 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" event={"ID":"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff","Type":"ContainerStarted","Data":"94bad7d2400b886fcfc124c0acd15d151b93daa002dc3bda50bda6cd45f8f7f4"} Nov 02 17:44:11.430967 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:11.430900 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podStartSLOduration=2.430880056 podStartE2EDuration="2.430880056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:11.427043964 +0000 UTC m=+616.118097976" watchObservedRunningTime="2026-11-02 17:44:11.430880056 +0000 UTC m=+616.121934069" Nov 02 17:44:12.149283 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:12.149234 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:20.149034 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:20.149000 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:20.163948 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:20.163908 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:20.399912 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:20.399829 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:44:22.918064 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:22.918014 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg"] Nov 02 17:44:24.507421 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:24.507388 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps"] Nov 02 17:44:32.528596 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.528560 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm"] Nov 02 17:44:32.533905 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.533879 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:32.544947 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.544919 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-24cfb6\"/\"distinct-w1-24cfb6-kserve-self-signed-certs\"" Nov 02 17:44:32.549014 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.548988 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm"] Nov 02 17:44:32.681599 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.681554 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-dshm\") pod \"distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:32.681821 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.681656 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxqwb\" (UniqueName: \"kubernetes.io/projected/3b3e335e-4328-429a-9433-728436384eba-kube-api-access-vxqwb\") pod \"distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:32.681821 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.681772 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-home\") pod \"distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:32.681996 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.681966 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-model-cache\") pod \"distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:32.682204 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.682182 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-tmp-dir\") pod \"distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:32.682395 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.682378 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3e335e-4328-429a-9433-728436384eba-tls-certs\") pod \"distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:32.784214 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.784117 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-dshm\") pod \"distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:32.784214 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.784181 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxqwb\" (UniqueName: \"kubernetes.io/projected/3b3e335e-4328-429a-9433-728436384eba-kube-api-access-vxqwb\") pod \"distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:32.784451 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.784289 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-home\") pod \"distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:32.784451 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.784391 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-model-cache\") pod \"distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:32.784451 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.784418 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-tmp-dir\") pod \"distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:32.784451 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.784443 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3e335e-4328-429a-9433-728436384eba-tls-certs\") pod \"distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:32.784765 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.784740 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-home\") pod \"distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:32.784832 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.784810 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-model-cache\") pod \"distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:32.784899 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.784858 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-tmp-dir\") pod \"distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:32.786570 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.786541 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-dshm\") pod \"distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:32.786876 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.786858 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3e335e-4328-429a-9433-728436384eba-tls-certs\") pod \"distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:32.799946 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.799924 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxqwb\" (UniqueName: \"kubernetes.io/projected/3b3e335e-4328-429a-9433-728436384eba-kube-api-access-vxqwb\") pod \"distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:32.848328 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:32.848285 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:33.138668 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:33.138569 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm"] Nov 02 17:44:33.141584 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:44:33.141546 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b3e335e_4328_429a_9433_728436384eba.slice/crio-9a78dcc22ed221b448cf21e8678b5f1fba2a3882d83e2f0f6d3d42b7ee2d1ea3 WatchSource:0}: Error finding container 9a78dcc22ed221b448cf21e8678b5f1fba2a3882d83e2f0f6d3d42b7ee2d1ea3: Status 404 returned error can't find the container with id 9a78dcc22ed221b448cf21e8678b5f1fba2a3882d83e2f0f6d3d42b7ee2d1ea3 Nov 02 17:44:33.469321 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:33.469285 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" event={"ID":"3b3e335e-4328-429a-9433-728436384eba","Type":"ContainerStarted","Data":"7f25e694f509f0931bf2bb9ff2aadd979adec71353a14975b62176efb4e320bf"} Nov 02 17:44:33.469321 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:33.469327 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" event={"ID":"3b3e335e-4328-429a-9433-728436384eba","Type":"ContainerStarted","Data":"9a78dcc22ed221b448cf21e8678b5f1fba2a3882d83e2f0f6d3d42b7ee2d1ea3"} Nov 02 17:44:33.494133 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:33.494081 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podStartSLOduration=1.4940668480000001 podStartE2EDuration="1.494066848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:33.491904314 +0000 UTC m=+638.182958327" watchObservedRunningTime="2026-11-02 17:44:33.494066848 +0000 UTC m=+638.185120890" Nov 02 17:44:33.850708 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:33.849734 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:34.531742 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:44:34.529651 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:44:35.405073 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.405036 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b"] Nov 02 17:44:35.518087 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.518049 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b"] Nov 02 17:44:35.520474 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.518388 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.548067 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.547799 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.548067 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.547852 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.548067 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.547886 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ck7w\" (UniqueName: \"kubernetes.io/projected/398ed64f-2082-405d-a5c3-3628b0fdd21f-kube-api-access-4ck7w\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.548067 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.547929 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398ed64f-2082-405d-a5c3-3628b0fdd21f-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.548067 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.547976 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.548067 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.548027 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.548067 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.548053 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.548485 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.548114 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.548485 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.548198 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.651722 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.649961 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.651722 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.650072 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.651722 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.650125 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.651722 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.650171 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.651722 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.650209 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ck7w\" (UniqueName: \"kubernetes.io/projected/398ed64f-2082-405d-a5c3-3628b0fdd21f-kube-api-access-4ck7w\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.651722 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.650262 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398ed64f-2082-405d-a5c3-3628b0fdd21f-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.651722 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.650296 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.651722 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.650362 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.651722 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.650393 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.651722 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.650852 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.654061 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.654039 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.655327 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.655277 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.655723 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.655701 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.656030 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.656007 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398ed64f-2082-405d-a5c3-3628b0fdd21f-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.702356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.702285 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.705783 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.704793 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.709697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.709658 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.731194 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.726667 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ck7w\" (UniqueName: \"kubernetes.io/projected/398ed64f-2082-405d-a5c3-3628b0fdd21f-kube-api-access-4ck7w\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:35.920357 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:35.919927 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:36.379276 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:36.379223 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b"] Nov 02 17:44:36.433032 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:36.432979 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:36.459095 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:36.459039 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:36.459416 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:36.459387 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:36.553048 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:36.553014 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" event={"ID":"398ed64f-2082-405d-a5c3-3628b0fdd21f","Type":"ContainerStarted","Data":"da5639a2e8eb3145b3261cee143643102375f573405cc9f49f447b1b57f5c470"} Nov 02 17:44:36.926639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:36.919030 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499"] Nov 02 17:44:36.976272 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:36.949942 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.021637 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.015676 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499"] Nov 02 17:44:37.050711 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.050668 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.051038 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.050963 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.051198 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.051180 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.051328 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.051311 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.051502 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.051433 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.113685 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.101120 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4qk8\" (UniqueName: \"kubernetes.io/projected/a15cd03a-9403-4912-a060-4ae6dab8baf8-kube-api-access-d4qk8\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.113685 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.101188 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.113685 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.101260 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15cd03a-9403-4912-a060-4ae6dab8baf8-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.113685 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.101309 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.224650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.217256 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.224953 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.220527 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.229725 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.225092 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.229929 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.229890 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.231119 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.231092 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4qk8\" (UniqueName: \"kubernetes.io/projected/a15cd03a-9403-4912-a060-4ae6dab8baf8-kube-api-access-d4qk8\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.231338 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.231304 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.232092 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.232071 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15cd03a-9403-4912-a060-4ae6dab8baf8-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.232335 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.232312 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.232568 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.232548 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.232767 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.232740 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.236642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.233364 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15cd03a-9403-4912-a060-4ae6dab8baf8-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.236642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.225424 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.236642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.233485 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.236642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.233790 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.316639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.316395 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.511694 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.496296 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4qk8\" (UniqueName: \"kubernetes.io/projected/a15cd03a-9403-4912-a060-4ae6dab8baf8-kube-api-access-d4qk8\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.511694 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.500562 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.511694 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.501150 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.562083 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.562043 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" event={"ID":"398ed64f-2082-405d-a5c3-3628b0fdd21f","Type":"ContainerStarted","Data":"8c6f42545013c742320559878b3b1e2dae9f592cd7e41915336ef23c5e656e7f"} Nov 02 17:44:37.589965 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.589907 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" podStartSLOduration=2.589885396 podStartE2EDuration="2.589885396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:37.586432997 +0000 UTC m=+642.277487010" watchObservedRunningTime="2026-11-02 17:44:37.589885396 +0000 UTC m=+642.280939409" Nov 02 17:44:37.634270 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.634222 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:37.921033 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.920998 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:37.923118 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.923089 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:44:37.923260 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.923147 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" podUID="398ed64f-2082-405d-a5c3-3628b0fdd21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:44:37.973933 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.972739 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499"] Nov 02 17:44:37.973933 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.973032 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" podUID="691f60d7-7f1b-4d75-9893-9060513dc151" containerName="main" containerID="cri-o://c061319a8fd6b45f16612f35a6414417ffc94e7caa2e470ecdd18205d7b37136" gracePeriod=45 Nov 02 17:44:37.982451 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.982384 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:37.982579 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.982517 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:37.982579 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:37.982560 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:38.454328 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.453968 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:44:38.570941 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.570838 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" event={"ID":"a15cd03a-9403-4912-a060-4ae6dab8baf8","Type":"ContainerStarted","Data":"d1afba8649d8a88565cd250391a5f4e01b6870473ca0819b56d2b8cf6c1e5d9e"} Nov 02 17:44:38.570941 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.570905 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" event={"ID":"a15cd03a-9403-4912-a060-4ae6dab8baf8","Type":"ContainerStarted","Data":"0d6323c67d2efb07b89c0c65b66f899ac7a4c6fa1e42fc2c6ab242fdd7f86464"} Nov 02 17:44:38.573113 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.573077 5601 generic.go:356] "Generic (PLEG): container finished" podID="691f60d7-7f1b-4d75-9893-9060513dc151" containerID="c061319a8fd6b45f16612f35a6414417ffc94e7caa2e470ecdd18205d7b37136" exitCode=0 Nov 02 17:44:38.573252 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.573135 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" Nov 02 17:44:38.573252 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.573163 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" event={"ID":"691f60d7-7f1b-4d75-9893-9060513dc151","Type":"ContainerDied","Data":"c061319a8fd6b45f16612f35a6414417ffc94e7caa2e470ecdd18205d7b37136"} Nov 02 17:44:38.573252 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.573232 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" event={"ID":"691f60d7-7f1b-4d75-9893-9060513dc151","Type":"ContainerDied","Data":"6acdd573fed341aa0ff61acba757b39f1dea320ce6c016082589290c64105874"} Nov 02 17:44:38.573252 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.573250 5601 scope.go:122] "RemoveContainer" containerID="c061319a8fd6b45f16612f35a6414417ffc94e7caa2e470ecdd18205d7b37136" Nov 02 17:44:38.584697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.584651 5601 scope.go:122] "RemoveContainer" containerID="c061319a8fd6b45f16612f35a6414417ffc94e7caa2e470ecdd18205d7b37136" Nov 02 17:44:38.584990 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:44:38.584964 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c061319a8fd6b45f16612f35a6414417ffc94e7caa2e470ecdd18205d7b37136\": container with ID starting with c061319a8fd6b45f16612f35a6414417ffc94e7caa2e470ecdd18205d7b37136 not found: ID does not exist" containerID="c061319a8fd6b45f16612f35a6414417ffc94e7caa2e470ecdd18205d7b37136" Nov 02 17:44:38.585055 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.585002 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c061319a8fd6b45f16612f35a6414417ffc94e7caa2e470ecdd18205d7b37136"} err="failed to get container status \"c061319a8fd6b45f16612f35a6414417ffc94e7caa2e470ecdd18205d7b37136\": rpc error: code = NotFound desc = could not find container \"c061319a8fd6b45f16612f35a6414417ffc94e7caa2e470ecdd18205d7b37136\": container with ID starting with c061319a8fd6b45f16612f35a6414417ffc94e7caa2e470ecdd18205d7b37136 not found: ID does not exist" Nov 02 17:44:38.591576 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.591551 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-dshm\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-dshm\") pod \"691f60d7-7f1b-4d75-9893-9060513dc151\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " Nov 02 17:44:38.591735 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.591596 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-tmp-dir\") pod \"691f60d7-7f1b-4d75-9893-9060513dc151\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " Nov 02 17:44:38.591735 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.591631 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-home\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-home\") pod \"691f60d7-7f1b-4d75-9893-9060513dc151\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " Nov 02 17:44:38.591735 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.591677 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/691f60d7-7f1b-4d75-9893-9060513dc151-tls-certs\" (UniqueName: \"kubernetes.io/secret/691f60d7-7f1b-4d75-9893-9060513dc151-tls-certs\") pod \"691f60d7-7f1b-4d75-9893-9060513dc151\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " Nov 02 17:44:38.591735 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.591728 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/691f60d7-7f1b-4d75-9893-9060513dc151-kube-api-access-b697h\" (UniqueName: \"kubernetes.io/projected/691f60d7-7f1b-4d75-9893-9060513dc151-kube-api-access-b697h\") pod \"691f60d7-7f1b-4d75-9893-9060513dc151\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " Nov 02 17:44:38.592017 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.591755 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-model-cache\") pod \"691f60d7-7f1b-4d75-9893-9060513dc151\" (UID: \"691f60d7-7f1b-4d75-9893-9060513dc151\") " Nov 02 17:44:38.592017 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.591955 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-tmp-dir" pod "691f60d7-7f1b-4d75-9893-9060513dc151" (UID: "691f60d7-7f1b-4d75-9893-9060513dc151"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:44:38.592248 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.592225 5601 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-tmp-dir\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:44:38.592248 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.592226 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-home" pod "691f60d7-7f1b-4d75-9893-9060513dc151" (UID: "691f60d7-7f1b-4d75-9893-9060513dc151"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:44:38.594054 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.592997 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" podStartSLOduration=2.592981594 podStartE2EDuration="2.592981594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:38.590293803 +0000 UTC m=+643.281347815" watchObservedRunningTime="2026-11-02 17:44:38.592981594 +0000 UTC m=+643.284035612" Nov 02 17:44:38.594054 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.593059 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-model-cache" pod "691f60d7-7f1b-4d75-9893-9060513dc151" (UID: "691f60d7-7f1b-4d75-9893-9060513dc151"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:44:38.594439 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.594415 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/691f60d7-7f1b-4d75-9893-9060513dc151-tls-certs" pod "691f60d7-7f1b-4d75-9893-9060513dc151" (UID: "691f60d7-7f1b-4d75-9893-9060513dc151"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:44:38.594527 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.594414 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-dshm" pod "691f60d7-7f1b-4d75-9893-9060513dc151" (UID: "691f60d7-7f1b-4d75-9893-9060513dc151"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:44:38.594609 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.594590 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691f60d7-7f1b-4d75-9893-9060513dc151-kube-api-access-b697h" pod "691f60d7-7f1b-4d75-9893-9060513dc151" (UID: "691f60d7-7f1b-4d75-9893-9060513dc151"). InnerVolumeSpecName "kube-api-access-b697h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:38.658639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.657703 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:38.693021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.692990 5601 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-home\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:44:38.693021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.693018 5601 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/691f60d7-7f1b-4d75-9893-9060513dc151-tls-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:44:38.693157 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.693030 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b697h\" (UniqueName: \"kubernetes.io/projected/691f60d7-7f1b-4d75-9893-9060513dc151-kube-api-access-b697h\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:44:38.693157 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.693040 5601 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-model-cache\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:44:38.693157 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.693049 5601 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/691f60d7-7f1b-4d75-9893-9060513dc151-dshm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:44:38.896988 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.896958 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg"] Nov 02 17:44:38.901157 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.901129 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg"] Nov 02 17:44:38.921251 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.921217 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:44:38.921390 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:38.921271 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" podUID="398ed64f-2082-405d-a5c3-3628b0fdd21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:44:39.282904 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:39.282761 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-b-f1a82c40-kserve-9d6ffb6f6-smxmg" podUID="691f60d7-7f1b-4d75-9893-9060513dc151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 02 17:44:39.425787 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:39.425745 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="691f60d7-7f1b-4d75-9893-9060513dc151" path="/var/lib/kubelet/pods/691f60d7-7f1b-4d75-9893-9060513dc151/volumes" Nov 02 17:44:39.565697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:39.558852 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" podUID="81707204-a73d-4ca8-bdc1-72995f633c36" containerName="main" containerID="cri-o://41d03f69322046a23a1f74ab0e42fc353b3e989ef907113b376e15137ec366d5" gracePeriod=45 Nov 02 17:44:39.655054 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:39.654448 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:39.655054 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:39.654515 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" podUID="a15cd03a-9403-4912-a060-4ae6dab8baf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 17:44:39.920792 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:39.920758 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:44:39.920963 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:39.920824 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" podUID="398ed64f-2082-405d-a5c3-3628b0fdd21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:44:40.010570 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.010545 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:44:40.107405 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.107341 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-tmp-dir\") pod \"81707204-a73d-4ca8-bdc1-72995f633c36\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " Nov 02 17:44:40.107553 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.107529 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-model-cache\") pod \"81707204-a73d-4ca8-bdc1-72995f633c36\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " Nov 02 17:44:40.107628 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.107579 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-home\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-home\") pod \"81707204-a73d-4ca8-bdc1-72995f633c36\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " Nov 02 17:44:40.107680 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.107632 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-tmp-dir" pod "81707204-a73d-4ca8-bdc1-72995f633c36" (UID: "81707204-a73d-4ca8-bdc1-72995f633c36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:44:40.107778 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.107759 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81707204-a73d-4ca8-bdc1-72995f633c36-kube-api-access-bk8ql\" (UniqueName: \"kubernetes.io/projected/81707204-a73d-4ca8-bdc1-72995f633c36-kube-api-access-bk8ql\") pod \"81707204-a73d-4ca8-bdc1-72995f633c36\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " Nov 02 17:44:40.107827 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.107810 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81707204-a73d-4ca8-bdc1-72995f633c36-tls-certs\" (UniqueName: \"kubernetes.io/secret/81707204-a73d-4ca8-bdc1-72995f633c36-tls-certs\") pod \"81707204-a73d-4ca8-bdc1-72995f633c36\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " Nov 02 17:44:40.107878 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.107862 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-dshm\") pod \"81707204-a73d-4ca8-bdc1-72995f633c36\" (UID: \"81707204-a73d-4ca8-bdc1-72995f633c36\") " Nov 02 17:44:40.107915 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.107872 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-model-cache" pod "81707204-a73d-4ca8-bdc1-72995f633c36" (UID: "81707204-a73d-4ca8-bdc1-72995f633c36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:44:40.107992 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.107967 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-home" pod "81707204-a73d-4ca8-bdc1-72995f633c36" (UID: "81707204-a73d-4ca8-bdc1-72995f633c36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:44:40.108422 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.108391 5601 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-model-cache\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:44:40.108422 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.108418 5601 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-home\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:44:40.108581 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.108432 5601 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-tmp-dir\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:44:40.110052 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.110028 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-dshm" pod "81707204-a73d-4ca8-bdc1-72995f633c36" (UID: "81707204-a73d-4ca8-bdc1-72995f633c36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:44:40.110178 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.110160 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81707204-a73d-4ca8-bdc1-72995f633c36-kube-api-access-bk8ql" pod "81707204-a73d-4ca8-bdc1-72995f633c36" (UID: "81707204-a73d-4ca8-bdc1-72995f633c36"). InnerVolumeSpecName "kube-api-access-bk8ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:40.124773 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.124743 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81707204-a73d-4ca8-bdc1-72995f633c36-tls-certs" pod "81707204-a73d-4ca8-bdc1-72995f633c36" (UID: "81707204-a73d-4ca8-bdc1-72995f633c36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:44:40.209155 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.209072 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk8ql\" (UniqueName: \"kubernetes.io/projected/81707204-a73d-4ca8-bdc1-72995f633c36-kube-api-access-bk8ql\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:44:40.209155 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.209104 5601 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81707204-a73d-4ca8-bdc1-72995f633c36-tls-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:44:40.209155 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.209114 5601 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81707204-a73d-4ca8-bdc1-72995f633c36-dshm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:44:40.599669 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.599571 5601 generic.go:356] "Generic (PLEG): container finished" podID="81707204-a73d-4ca8-bdc1-72995f633c36" containerID="41d03f69322046a23a1f74ab0e42fc353b3e989ef907113b376e15137ec366d5" exitCode=0 Nov 02 17:44:40.599669 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.599643 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" Nov 02 17:44:40.599669 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.599654 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" event={"ID":"81707204-a73d-4ca8-bdc1-72995f633c36","Type":"ContainerDied","Data":"41d03f69322046a23a1f74ab0e42fc353b3e989ef907113b376e15137ec366d5"} Nov 02 17:44:40.599883 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.599717 5601 scope.go:122] "RemoveContainer" containerID="41d03f69322046a23a1f74ab0e42fc353b3e989ef907113b376e15137ec366d5" Nov 02 17:44:40.599883 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.599808 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" event={"ID":"81707204-a73d-4ca8-bdc1-72995f633c36","Type":"ContainerDied","Data":"c0babf813abdb35095ce34ff5e7aac05ced1c55a776ed75c78bee8f58553e593"} Nov 02 17:44:40.610610 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.610582 5601 scope.go:122] "RemoveContainer" containerID="41d03f69322046a23a1f74ab0e42fc353b3e989ef907113b376e15137ec366d5" Nov 02 17:44:40.611365 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:44:40.611319 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d03f69322046a23a1f74ab0e42fc353b3e989ef907113b376e15137ec366d5\": container with ID starting with 41d03f69322046a23a1f74ab0e42fc353b3e989ef907113b376e15137ec366d5 not found: ID does not exist" containerID="41d03f69322046a23a1f74ab0e42fc353b3e989ef907113b376e15137ec366d5" Nov 02 17:44:40.611537 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.611377 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d03f69322046a23a1f74ab0e42fc353b3e989ef907113b376e15137ec366d5"} err="failed to get container status \"41d03f69322046a23a1f74ab0e42fc353b3e989ef907113b376e15137ec366d5\": rpc error: code = NotFound desc = could not find container \"41d03f69322046a23a1f74ab0e42fc353b3e989ef907113b376e15137ec366d5\": container with ID starting with 41d03f69322046a23a1f74ab0e42fc353b3e989ef907113b376e15137ec366d5 not found: ID does not exist" Nov 02 17:44:40.632222 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.632198 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps"] Nov 02 17:44:40.647643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.642137 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:40.647643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.642209 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" podUID="a15cd03a-9403-4912-a060-4ae6dab8baf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 17:44:40.688652 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.683327 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps"] Nov 02 17:44:40.920824 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.920786 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:44:40.920991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.920861 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" podUID="398ed64f-2082-405d-a5c3-3628b0fdd21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:44:40.927742 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:40.927710 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/e2e-conflict-a-f1a82c40-kserve-7cd9b54f7d-f47ps" podUID="81707204-a73d-4ca8-bdc1-72995f633c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 02 17:44:41.425536 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:41.425501 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81707204-a73d-4ca8-bdc1-72995f633c36" path="/var/lib/kubelet/pods/81707204-a73d-4ca8-bdc1-72995f633c36/volumes" Nov 02 17:44:41.647714 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:41.637940 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:41.647714 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:41.638006 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" podUID="a15cd03a-9403-4912-a060-4ae6dab8baf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 17:44:41.921407 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:41.921378 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:44:41.921817 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:41.921429 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" podUID="398ed64f-2082-405d-a5c3-3628b0fdd21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:44:42.638403 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:42.637980 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:42.638403 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:42.638040 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" podUID="a15cd03a-9403-4912-a060-4ae6dab8baf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 17:44:42.853639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:42.852977 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:42.893917 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:42.893836 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:42.922442 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:42.922036 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:44:42.922442 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:42.922117 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" podUID="398ed64f-2082-405d-a5c3-3628b0fdd21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:44:43.631571 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:43.631541 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:44:43.644411 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:43.644382 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:43.644676 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:43.644649 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" podUID="a15cd03a-9403-4912-a060-4ae6dab8baf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 17:44:43.920744 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:43.920715 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:44:43.920902 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:43.920766 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" podUID="398ed64f-2082-405d-a5c3-3628b0fdd21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:44:44.680640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:44.675809 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:44:44.680640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:44.675864 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" podUID="a15cd03a-9403-4912-a060-4ae6dab8baf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 17:44:45.304991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:45.304956 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:45.307569 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:45.307546 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:45.318320 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:45.318290 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 17:44:46.787807 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:46.787764 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:46.789571 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:46.789261 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" podUID="a15cd03a-9403-4912-a060-4ae6dab8baf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:46.821567 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:46.821526 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-jv8f4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:46.821872 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:46.821841 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" podUID="c32fa6f3-9665-4b3b-b093-22435190829b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.022640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:47.021796 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-wtks5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:47.022640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:47.021895 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" podUID="63f70366-70f1-475a-8e45-eb6da6b4828e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.819329 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:47.819291 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:47.819974 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:47.819935 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" podUID="a15cd03a-9403-4912-a060-4ae6dab8baf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.821225 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:47.821203 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:47.821592 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:47.821572 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:47.822303 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:47.822284 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 17:44:50.727483 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.727449 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp"] Nov 02 17:44:50.727966 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.727951 5601 cpu_manager.go:414] "containerMap: removing container" podUID="81707204-a73d-4ca8-bdc1-72995f633c36" containerName="main" Nov 02 17:44:50.727966 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.727967 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81707204-a73d-4ca8-bdc1-72995f633c36" containerName="main" Nov 02 17:44:50.728046 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.727991 5601 cpu_manager.go:414] "containerMap: removing container" podUID="691f60d7-7f1b-4d75-9893-9060513dc151" containerName="main" Nov 02 17:44:50.728046 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.727998 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="691f60d7-7f1b-4d75-9893-9060513dc151" containerName="main" Nov 02 17:44:50.728104 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.728064 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="81707204-a73d-4ca8-bdc1-72995f633c36" containerName="main" Nov 02 17:44:50.728104 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.728075 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="691f60d7-7f1b-4d75-9893-9060513dc151" containerName="main" Nov 02 17:44:50.733039 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.732884 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.737579 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.737556 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c1c9ac-b512-462a-961e-93ac960cde22-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.737692 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.737590 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wfkf\" (UniqueName: \"kubernetes.io/projected/62c1c9ac-b512-462a-961e-93ac960cde22-kube-api-access-4wfkf\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.737733 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.737710 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.737809 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.737740 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c1c9ac-b512-462a-961e-93ac960cde22-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.737809 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.737761 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.737809 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.737781 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.737978 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.737903 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c1c9ac-b512-462a-961e-93ac960cde22-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.738035 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.737984 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.738069 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.738050 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.757290 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.757244 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2"] Nov 02 17:44:50.764202 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.763398 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.764202 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.763401 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp"] Nov 02 17:44:50.791570 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.791537 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2"] Nov 02 17:44:50.838921 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.838886 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.839109 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.838946 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.839109 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.839032 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c1c9ac-b512-462a-961e-93ac960cde22-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.839109 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.839075 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.839109 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.839099 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.839307 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.839165 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.839307 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.839204 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c1c9ac-b512-462a-961e-93ac960cde22-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.839307 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.839237 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.839307 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.839292 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.839588 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.839323 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.839588 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.839407 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.839588 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.839478 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c1c9ac-b512-462a-961e-93ac960cde22-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.839588 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.839517 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wfkf\" (UniqueName: \"kubernetes.io/projected/62c1c9ac-b512-462a-961e-93ac960cde22-kube-api-access-4wfkf\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.839588 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.839531 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.839588 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.839547 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.839870 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.839587 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkk5x\" (UniqueName: \"kubernetes.io/projected/a268dccb-3642-470f-b0b2-d9e4c4257ca5-kube-api-access-zkk5x\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.839870 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.839655 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.839870 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.839760 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.839870 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.839783 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.839870 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.839796 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.839870 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.839848 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.840110 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.840008 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.840347 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.840319 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c1c9ac-b512-462a-961e-93ac960cde22-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.842202 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.842096 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.842202 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.842122 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c1c9ac-b512-462a-961e-93ac960cde22-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.848081 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.848053 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c1c9ac-b512-462a-961e-93ac960cde22-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.848462 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.848438 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wfkf\" (UniqueName: \"kubernetes.io/projected/62c1c9ac-b512-462a-961e-93ac960cde22-kube-api-access-4wfkf\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:50.940948 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.940911 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.940948 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.940959 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.941146 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.941067 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.941146 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.941092 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkk5x\" (UniqueName: \"kubernetes.io/projected/a268dccb-3642-470f-b0b2-d9e4c4257ca5-kube-api-access-zkk5x\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.941222 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.941154 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.941222 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.941184 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.941222 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.941206 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.941373 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.941260 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.941373 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.941342 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.941373 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.941358 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.941523 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.941505 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.941821 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.941795 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.941957 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.941827 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.942098 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.942069 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.943515 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.943493 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.943884 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.943854 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.950207 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.950181 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:50.950468 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:50.950440 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkk5x\" (UniqueName: \"kubernetes.io/projected/a268dccb-3642-470f-b0b2-d9e4c4257ca5-kube-api-access-zkk5x\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:51.053800 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:51.053719 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:51.093987 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:51.093962 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:51.341469 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:51.341441 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp"] Nov 02 17:44:51.343075 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:44:51.343042 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c1c9ac_b512_462a_961e_93ac960cde22.slice/crio-658c555c084846031c76367302610fa8bba0ad443acd053d537705be7673b1ce WatchSource:0}: Error finding container 658c555c084846031c76367302610fa8bba0ad443acd053d537705be7673b1ce: Status 404 returned error can't find the container with id 658c555c084846031c76367302610fa8bba0ad443acd053d537705be7673b1ce Nov 02 17:44:51.345540 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:51.345508 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:51.345656 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:51.345576 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:51.345656 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:51.345606 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:51.402570 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:51.402545 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2"] Nov 02 17:44:51.412091 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:51.412050 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:51.412196 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:51.412144 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:51.412196 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:51.412187 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:51.827354 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:51.827316 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" event={"ID":"a268dccb-3642-470f-b0b2-d9e4c4257ca5","Type":"ContainerStarted","Data":"d86f80539de6b2f21a64b823dd2c3ef3e7d30ebd7c698969905b7eb394f46ecf"} Nov 02 17:44:51.827813 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:51.827376 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" event={"ID":"a268dccb-3642-470f-b0b2-d9e4c4257ca5","Type":"ContainerStarted","Data":"ec4394472b6598c599ab16c634e4d39fc2acccec89b21b08f1977d116d4c3cce"} Nov 02 17:44:51.829409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:51.829379 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" event={"ID":"62c1c9ac-b512-462a-961e-93ac960cde22","Type":"ContainerStarted","Data":"2a3d034061e431cb778d461f53b91f9fdc6cda3de8561f245df9a0baa26ab552"} Nov 02 17:44:51.829557 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:51.829429 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" event={"ID":"62c1c9ac-b512-462a-961e-93ac960cde22","Type":"ContainerStarted","Data":"658c555c084846031c76367302610fa8bba0ad443acd053d537705be7673b1ce"} Nov 02 17:44:51.873128 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:51.873073 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" podStartSLOduration=1.873055436 podStartE2EDuration="1.873055436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:51.870937203 +0000 UTC m=+656.561991221" watchObservedRunningTime="2026-11-02 17:44:51.873055436 +0000 UTC m=+656.564109450" Nov 02 17:44:51.901105 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:51.901057 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" podStartSLOduration=1.901043778 podStartE2EDuration="1.901043778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:51.897823507 +0000 UTC m=+656.588877522" watchObservedRunningTime="2026-11-02 17:44:51.901043778 +0000 UTC m=+656.592097851" Nov 02 17:44:51.990506 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:51.990478 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55"] Nov 02 17:44:51.995019 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:51.994995 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.026296 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.026257 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55"] Nov 02 17:44:52.054349 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.054323 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:44:52.055597 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.055572 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:52.055743 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.055632 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" podUID="62c1c9ac-b512-462a-961e-93ac960cde22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 17:44:52.056177 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.056153 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.056240 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.056222 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.056295 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.056256 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfkzx\" (UniqueName: \"kubernetes.io/projected/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-kube-api-access-rfkzx\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.056347 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.056301 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.056399 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.056383 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.056452 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.056421 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.056504 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.056465 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.056504 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.056491 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.056602 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.056525 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.095016 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.094952 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:44:52.096372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.096348 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:52.096481 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.096393 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" podUID="a268dccb-3642-470f-b0b2-d9e4c4257ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 17:44:52.157921 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.157890 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.158016 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.157958 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.158016 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.157993 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.158125 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.158020 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.158125 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.158036 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.158125 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.158070 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.158277 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.158171 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.158277 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.158226 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.158277 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.158253 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfkzx\" (UniqueName: \"kubernetes.io/projected/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-kube-api-access-rfkzx\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.158493 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.158451 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.158631 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.158591 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.158712 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.158676 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.158775 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.158755 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.158833 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.158816 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.160842 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.160815 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.161005 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.160990 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.167584 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.167557 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfkzx\" (UniqueName: \"kubernetes.io/projected/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-kube-api-access-rfkzx\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.168450 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.168427 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.313072 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.313024 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:52.698252 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.698155 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55"] Nov 02 17:44:52.715599 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.715561 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:52.715756 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.715679 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:52.715756 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.715712 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:44:52.879053 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.878352 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" event={"ID":"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda","Type":"ContainerStarted","Data":"d6e9b42e6e1a13e7be3b5452aebc2a5471750bdf97e4c74b20d6371ad80778ee"} Nov 02 17:44:52.930642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.923506 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd"] Nov 02 17:44:52.969981 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.969902 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd"] Nov 02 17:44:52.970306 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:52.970244 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:53.010667 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.000974 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-24cfb6\"/\"distinct-2-w1-24cfb6-kserve-self-signed-certs\"" Nov 02 17:44:53.028970 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.027393 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmv84\" (UniqueName: \"kubernetes.io/projected/5d064f41-fece-402e-a07f-8515424b2c79-kube-api-access-nmv84\") pod \"distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:53.028970 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.027487 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-home\") pod \"distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:53.028970 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.027525 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d064f41-fece-402e-a07f-8515424b2c79-tls-certs\") pod \"distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:53.042682 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.036768 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-model-cache\") pod \"distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:53.042682 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.036874 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-dshm\") pod \"distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:53.042682 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.037047 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-tmp-dir\") pod \"distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:53.071639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.055058 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:53.071639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.055114 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" podUID="62c1c9ac-b512-462a-961e-93ac960cde22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 17:44:53.108250 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.107474 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:53.108250 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.107539 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" podUID="a268dccb-3642-470f-b0b2-d9e4c4257ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 17:44:53.144977 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.142453 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-dshm\") pod \"distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:53.144977 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.142599 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-tmp-dir\") pod \"distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:53.144977 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.142682 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmv84\" (UniqueName: \"kubernetes.io/projected/5d064f41-fece-402e-a07f-8515424b2c79-kube-api-access-nmv84\") pod \"distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:53.144977 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.142754 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-home\") pod \"distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:53.144977 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.142786 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d064f41-fece-402e-a07f-8515424b2c79-tls-certs\") pod \"distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:53.144977 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.142870 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-model-cache\") pod \"distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:53.144977 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.143373 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-model-cache\") pod \"distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:53.163683 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.146172 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-tmp-dir\") pod \"distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:53.163683 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.146416 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-home\") pod \"distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:53.164466 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.164437 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-dshm\") pod \"distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:53.168691 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.164890 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmv84\" (UniqueName: \"kubernetes.io/projected/5d064f41-fece-402e-a07f-8515424b2c79-kube-api-access-nmv84\") pod \"distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:53.185640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.182474 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d064f41-fece-402e-a07f-8515424b2c79-tls-certs\") pod \"distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:53.384776 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:53.384681 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:54.020643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:54.018105 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" event={"ID":"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda","Type":"ContainerStarted","Data":"82fa031de84b14f2d407f7d5b7d75fb892999bff35e8220916dd4fd1b0f5c856"} Nov 02 17:44:54.079642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:54.078609 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" podStartSLOduration=3.078589061 podStartE2EDuration="3.078589061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:54.054074554 +0000 UTC m=+658.745128593" watchObservedRunningTime="2026-11-02 17:44:54.078589061 +0000 UTC m=+658.769643073" Nov 02 17:44:54.123832 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:54.123746 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:54.124091 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:54.124058 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" podUID="a268dccb-3642-470f-b0b2-d9e4c4257ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 17:44:54.164640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:54.164404 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:54.164640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:54.164466 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" podUID="62c1c9ac-b512-462a-961e-93ac960cde22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 17:44:54.247567 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:54.247526 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd"] Nov 02 17:44:54.317642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:54.315141 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:44:54.317642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:54.317462 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:44:54.317642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:54.317514 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" podUID="ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 17:44:55.109710 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:55.092850 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:55.109710 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:55.092911 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" podUID="62c1c9ac-b512-462a-961e-93ac960cde22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 17:44:55.109710 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:55.094335 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:55.109710 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:55.094378 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" podUID="a268dccb-3642-470f-b0b2-d9e4c4257ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 17:44:55.179649 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:55.175862 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" event={"ID":"5d064f41-fece-402e-a07f-8515424b2c79","Type":"ContainerStarted","Data":"cc1a453576a5fde1729406584f452fbe66a04e2f16db4c4eb504463a7cc9415d"} Nov 02 17:44:55.315483 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:55.315012 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:44:55.315483 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:55.315091 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" podUID="ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 17:44:56.059640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:56.059019 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:56.059640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:56.059079 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" podUID="62c1c9ac-b512-462a-961e-93ac960cde22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 17:44:56.105641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:56.101893 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:56.105641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:56.101949 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" podUID="a268dccb-3642-470f-b0b2-d9e4c4257ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 17:44:56.215638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:56.215291 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" event={"ID":"5d064f41-fece-402e-a07f-8515424b2c79","Type":"ContainerStarted","Data":"ebadcc1635e2bd3139ffe8d9b2ec38c00de6651b8a862d7fc92e5ebf0a9bed96"} Nov 02 17:44:56.311693 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:56.309305 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podStartSLOduration=4.309283727 podStartE2EDuration="4.309283727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:56.295993656 +0000 UTC m=+660.987047669" watchObservedRunningTime="2026-11-02 17:44:56.309283727 +0000 UTC m=+661.000337744" Nov 02 17:44:56.315639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:56.314157 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:44:56.315639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:56.314226 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" podUID="ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 17:44:56.387285 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:56.386586 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:44:57.057175 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:57.057137 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:57.057503 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:57.057458 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" podUID="62c1c9ac-b512-462a-961e-93ac960cde22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 17:44:57.100217 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:57.100177 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:57.100499 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:57.100469 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" podUID="a268dccb-3642-470f-b0b2-d9e4c4257ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 17:44:57.313978 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:57.313885 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:44:57.314571 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:57.314536 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" podUID="ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 17:44:58.058635 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:58.058145 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:58.058635 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:58.058223 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" podUID="62c1c9ac-b512-462a-961e-93ac960cde22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 17:44:58.098050 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:58.098011 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:58.098352 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:58.098316 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" podUID="a268dccb-3642-470f-b0b2-d9e4c4257ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 17:44:58.314691 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:58.314545 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:44:58.315180 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:58.314693 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" podUID="ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 17:44:59.056636 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:59.056018 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 17:44:59.056636 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:59.056087 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" podUID="62c1c9ac-b512-462a-961e-93ac960cde22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 17:44:59.095338 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:59.095300 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 17:44:59.095533 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:59.095362 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" podUID="a268dccb-3642-470f-b0b2-d9e4c4257ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 17:44:59.314305 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:59.314206 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:44:59.314495 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:44:59.314297 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" podUID="ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 17:45:00.181529 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:00.181484 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q"] Nov 02 17:45:00.187804 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:00.187773 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" Nov 02 17:45:00.214187 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:00.214060 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q"] Nov 02 17:45:00.291074 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:00.291035 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\"" Nov 02 17:45:00.295018 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:00.294987 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q\" (UID: \"d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" Nov 02 17:45:00.295118 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:00.295056 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz9rh\" (UniqueName: \"kubernetes.io/projected/d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af-kube-api-access-nz9rh\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q\" (UID: \"d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" Nov 02 17:45:00.314551 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:00.314517 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 17:45:00.314723 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:00.314581 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" podUID="ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 17:45:00.399276 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:00.399232 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q\" (UID: \"d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" Nov 02 17:45:00.399459 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:00.399313 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz9rh\" (UniqueName: \"kubernetes.io/projected/d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af-kube-api-access-nz9rh\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q\" (UID: \"d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" Nov 02 17:45:00.401433 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:00.401403 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q\" (UID: \"d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" Nov 02 17:45:00.411662 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:00.411606 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz9rh\" (UniqueName: \"kubernetes.io/projected/d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af-kube-api-access-nz9rh\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q\" (UID: \"d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" Nov 02 17:45:00.524982 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:00.524886 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" Nov 02 17:45:01.026672 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:01.025918 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q"] Nov 02 17:45:01.031659 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:45:01.029683 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f3d06b_3a84_4f22_90e8_3fce0dcdd3af.slice/crio-ff96cb2e8569b7c199458e61f81a6854303d9594afd9233a78cea6a503addb59 WatchSource:0}: Error finding container ff96cb2e8569b7c199458e61f81a6854303d9594afd9233a78cea6a503addb59: Status 404 returned error can't find the container with id ff96cb2e8569b7c199458e61f81a6854303d9594afd9233a78cea6a503addb59 Nov 02 17:45:01.098252 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:01.095758 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:45:01.098252 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:01.095831 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" podUID="62c1c9ac-b512-462a-961e-93ac960cde22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:01.147647 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:01.136713 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:45:01.147647 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:01.136806 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" podUID="a268dccb-3642-470f-b0b2-d9e4c4257ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:01.268290 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:01.268244 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" event={"ID":"d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af","Type":"ContainerStarted","Data":"ff96cb2e8569b7c199458e61f81a6854303d9594afd9233a78cea6a503addb59"} Nov 02 17:45:01.321569 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:01.321469 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:45:01.322521 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:01.322497 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:45:01.323329 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:01.323304 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 17:45:01.387319 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:01.387286 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:45:01.388453 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:01.388432 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:45:01.402985 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:01.402958 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 17:45:01.452608 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:01.451578 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:45:01.452608 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:01.452493 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:45:01.497116 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:01.497086 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:45:03.392694 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:03.388145 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:45:03.481642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:03.481091 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:45:04.532685 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:04.532374 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:45:04.538142 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:45:04.537050 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:45:05.443511 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.443479 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd"] Nov 02 17:45:05.453994 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.453932 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.457152 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.457124 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd"] Nov 02 17:45:05.608474 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.608442 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.608910 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.608497 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.608910 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.608734 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.608910 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.608783 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e162853a-15f2-42ad-b350-bbc126c32272-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.608910 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.608807 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e162853a-15f2-42ad-b350-bbc126c32272-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.609106 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.608911 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e162853a-15f2-42ad-b350-bbc126c32272-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.609106 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.608977 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.609106 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.609042 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.609106 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.609072 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p45p8\" (UniqueName: \"kubernetes.io/projected/e162853a-15f2-42ad-b350-bbc126c32272-kube-api-access-p45p8\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.710320 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.710240 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e162853a-15f2-42ad-b350-bbc126c32272-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.710320 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.710297 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.710553 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.710343 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.710553 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.710370 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p45p8\" (UniqueName: \"kubernetes.io/projected/e162853a-15f2-42ad-b350-bbc126c32272-kube-api-access-p45p8\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.710553 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.710426 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.710553 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.710465 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.710783 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.710630 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.710783 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.710664 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e162853a-15f2-42ad-b350-bbc126c32272-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.710783 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.710687 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e162853a-15f2-42ad-b350-bbc126c32272-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.710783 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.710768 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.711003 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.710800 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.711072 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.711057 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.711190 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.711164 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e162853a-15f2-42ad-b350-bbc126c32272-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.711249 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.711197 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.713110 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.713066 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.713382 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.713363 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e162853a-15f2-42ad-b350-bbc126c32272-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.721152 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.721121 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e162853a-15f2-42ad-b350-bbc126c32272-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.721435 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.721411 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p45p8\" (UniqueName: \"kubernetes.io/projected/e162853a-15f2-42ad-b350-bbc126c32272-kube-api-access-p45p8\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:05.774935 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:05.774647 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:06.048171 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.048090 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m"] Nov 02 17:45:06.051917 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.051894 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m" Nov 02 17:45:06.056062 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.056032 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:45:06.060331 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.060307 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m"] Nov 02 17:45:06.073197 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.073169 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd"] Nov 02 17:45:06.107321 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:45:06.107293 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode162853a_15f2_42ad_b350_bbc126c32272.slice/crio-eb7f0a62777c022a45b7c70260defb6f840a9b05fb4c584e4bcf829f21d26472 WatchSource:0}: Error finding container eb7f0a62777c022a45b7c70260defb6f840a9b05fb4c584e4bcf829f21d26472: Status 404 returned error can't find the container with id eb7f0a62777c022a45b7c70260defb6f840a9b05fb4c584e4bcf829f21d26472 Nov 02 17:45:06.110738 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.110700 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:06.110824 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.110784 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:06.110864 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.110827 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:06.218322 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.218286 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5177fbbe-15f1-4b55-84dd-1f2d2a952617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e08dec13-d8x8m\" (UID: \"5177fbbe-15f1-4b55-84dd-1f2d2a952617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m" Nov 02 17:45:06.218558 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.218534 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plmwf\" (UniqueName: \"kubernetes.io/projected/5177fbbe-15f1-4b55-84dd-1f2d2a952617-kube-api-access-plmwf\") pod \"maas-api-revoke-sub-e08dec13-d8x8m\" (UID: \"5177fbbe-15f1-4b55-84dd-1f2d2a952617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m" Nov 02 17:45:06.218691 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.218676 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5177fbbe-15f1-4b55-84dd-1f2d2a952617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e08dec13-d8x8m\" (UID: \"5177fbbe-15f1-4b55-84dd-1f2d2a952617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m" Nov 02 17:45:06.320239 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.320148 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5177fbbe-15f1-4b55-84dd-1f2d2a952617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e08dec13-d8x8m\" (UID: \"5177fbbe-15f1-4b55-84dd-1f2d2a952617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m" Nov 02 17:45:06.320428 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.320236 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5177fbbe-15f1-4b55-84dd-1f2d2a952617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e08dec13-d8x8m\" (UID: \"5177fbbe-15f1-4b55-84dd-1f2d2a952617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m" Nov 02 17:45:06.320428 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.320403 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plmwf\" (UniqueName: \"kubernetes.io/projected/5177fbbe-15f1-4b55-84dd-1f2d2a952617-kube-api-access-plmwf\") pod \"maas-api-revoke-sub-e08dec13-d8x8m\" (UID: \"5177fbbe-15f1-4b55-84dd-1f2d2a952617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m" Nov 02 17:45:06.321238 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.321101 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5177fbbe-15f1-4b55-84dd-1f2d2a952617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e08dec13-d8x8m\" (UID: \"5177fbbe-15f1-4b55-84dd-1f2d2a952617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m" Nov 02 17:45:06.322981 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.322955 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5177fbbe-15f1-4b55-84dd-1f2d2a952617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e08dec13-d8x8m\" (UID: \"5177fbbe-15f1-4b55-84dd-1f2d2a952617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m" Nov 02 17:45:06.329449 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.329422 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plmwf\" (UniqueName: \"kubernetes.io/projected/5177fbbe-15f1-4b55-84dd-1f2d2a952617-kube-api-access-plmwf\") pod \"maas-api-revoke-sub-e08dec13-d8x8m\" (UID: \"5177fbbe-15f1-4b55-84dd-1f2d2a952617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m" Nov 02 17:45:06.374762 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.374727 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m" Nov 02 17:45:06.404984 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.404943 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" event={"ID":"e162853a-15f2-42ad-b350-bbc126c32272","Type":"ContainerStarted","Data":"e629be86004f42bd7130d861f15eb0e0dbd3508c75fbede6e774784cd052e409"} Nov 02 17:45:06.405160 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.405024 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" event={"ID":"e162853a-15f2-42ad-b350-bbc126c32272","Type":"ContainerStarted","Data":"eb7f0a62777c022a45b7c70260defb6f840a9b05fb4c584e4bcf829f21d26472"} Nov 02 17:45:06.431124 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.431065 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" podStartSLOduration=1.431043909 podStartE2EDuration="1.431043909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:06.426285423 +0000 UTC m=+671.117339450" watchObservedRunningTime="2026-11-02 17:45:06.431043909 +0000 UTC m=+671.122097925" Nov 02 17:45:06.681410 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.681368 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m"] Nov 02 17:45:06.684604 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:45:06.684573 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5177fbbe_15f1_4b55_84dd_1f2d2a952617.slice/crio-353d69563b3b5a453042c9140cc96baa812f49f09eeb01b09dff4975c79948d2 WatchSource:0}: Error finding container 353d69563b3b5a453042c9140cc96baa812f49f09eeb01b09dff4975c79948d2: Status 404 returned error can't find the container with id 353d69563b3b5a453042c9140cc96baa812f49f09eeb01b09dff4975c79948d2 Nov 02 17:45:06.775889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:06.775851 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:07.412532 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:07.412448 5601 generic.go:356] "Generic (PLEG): container finished" podID="d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af" containerID="2e9c84e9e6daf2acc06a66cef7031ca463327b9e768fcde345a55a391173e6c6" exitCode=60 Nov 02 17:45:07.412707 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:07.412531 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" event={"ID":"d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af","Type":"ContainerDied","Data":"2e9c84e9e6daf2acc06a66cef7031ca463327b9e768fcde345a55a391173e6c6"} Nov 02 17:45:07.412824 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:07.412809 5601 scope.go:122] "RemoveContainer" containerID="2e9c84e9e6daf2acc06a66cef7031ca463327b9e768fcde345a55a391173e6c6" Nov 02 17:45:07.414336 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:07.414310 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m" event={"ID":"5177fbbe-15f1-4b55-84dd-1f2d2a952617","Type":"ContainerStarted","Data":"3850531e248af63bbe4753a7e34aa8a09598ceb62ed2979fda57ee2d080e96ab"} Nov 02 17:45:07.414467 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:07.414360 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m" event={"ID":"5177fbbe-15f1-4b55-84dd-1f2d2a952617","Type":"ContainerStarted","Data":"353d69563b3b5a453042c9140cc96baa812f49f09eeb01b09dff4975c79948d2"} Nov 02 17:45:07.777022 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:07.776988 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 17:45:07.777390 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:07.777042 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" podUID="e162853a-15f2-42ad-b350-bbc126c32272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 17:45:08.422018 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:08.421980 5601 generic.go:356] "Generic (PLEG): container finished" podID="5177fbbe-15f1-4b55-84dd-1f2d2a952617" containerID="3850531e248af63bbe4753a7e34aa8a09598ceb62ed2979fda57ee2d080e96ab" exitCode=0 Nov 02 17:45:08.422218 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:08.422052 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m" event={"ID":"5177fbbe-15f1-4b55-84dd-1f2d2a952617","Type":"ContainerDied","Data":"3850531e248af63bbe4753a7e34aa8a09598ceb62ed2979fda57ee2d080e96ab"} Nov 02 17:45:08.424330 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:08.424302 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" event={"ID":"d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af","Type":"ContainerStarted","Data":"52b1079b203f2bb6bd427228092f1d93dd51399ba049f9bc51654769dda07637"} Nov 02 17:45:08.480392 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:08.480335 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" podStartSLOduration=3.392033854 podStartE2EDuration="8.48031899s" totalImagesPullingTime="5.088285136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:08.478587519 +0000 UTC m=+673.169641528" watchObservedRunningTime="2026-11-02 17:45:08.48031899 +0000 UTC m=+673.171373001" Nov 02 17:45:08.775856 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:08.775771 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 17:45:08.775856 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:08.775841 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" podUID="e162853a-15f2-42ad-b350-bbc126c32272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 17:45:09.462423 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:09.462384 5601 generic.go:356] "Generic (PLEG): container finished" podID="d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af" containerID="52b1079b203f2bb6bd427228092f1d93dd51399ba049f9bc51654769dda07637" exitCode=60 Nov 02 17:45:09.464501 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:09.464475 5601 scope.go:122] "RemoveContainer" containerID="52b1079b203f2bb6bd427228092f1d93dd51399ba049f9bc51654769dda07637" Nov 02 17:45:09.464863 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:45:09.464832 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q_odh-ai-gateway-infra(d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" podUID="d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af" Nov 02 17:45:09.466073 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:09.466043 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" event={"ID":"d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af","Type":"ContainerDied","Data":"52b1079b203f2bb6bd427228092f1d93dd51399ba049f9bc51654769dda07637"} Nov 02 17:45:09.466168 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:09.466107 5601 scope.go:122] "RemoveContainer" containerID="2e9c84e9e6daf2acc06a66cef7031ca463327b9e768fcde345a55a391173e6c6" Nov 02 17:45:09.683189 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:09.683157 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m" Nov 02 17:45:09.775338 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:09.775253 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 17:45:09.775338 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:09.775312 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" podUID="e162853a-15f2-42ad-b350-bbc126c32272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 17:45:09.795583 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:09.795550 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5177fbbe-15f1-4b55-84dd-1f2d2a952617-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5177fbbe-15f1-4b55-84dd-1f2d2a952617-maas-api-openshift-ca\") pod \"5177fbbe-15f1-4b55-84dd-1f2d2a952617\" (UID: \"5177fbbe-15f1-4b55-84dd-1f2d2a952617\") " Nov 02 17:45:09.795769 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:09.795658 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5177fbbe-15f1-4b55-84dd-1f2d2a952617-kube-api-access-plmwf\" (UniqueName: \"kubernetes.io/projected/5177fbbe-15f1-4b55-84dd-1f2d2a952617-kube-api-access-plmwf\") pod \"5177fbbe-15f1-4b55-84dd-1f2d2a952617\" (UID: \"5177fbbe-15f1-4b55-84dd-1f2d2a952617\") " Nov 02 17:45:09.795769 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:09.795724 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5177fbbe-15f1-4b55-84dd-1f2d2a952617-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5177fbbe-15f1-4b55-84dd-1f2d2a952617-maas-api-xks-ca\") pod \"5177fbbe-15f1-4b55-84dd-1f2d2a952617\" (UID: \"5177fbbe-15f1-4b55-84dd-1f2d2a952617\") " Nov 02 17:45:09.795969 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:09.795942 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5177fbbe-15f1-4b55-84dd-1f2d2a952617-maas-api-openshift-ca" pod "5177fbbe-15f1-4b55-84dd-1f2d2a952617" (UID: "5177fbbe-15f1-4b55-84dd-1f2d2a952617"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:09.796212 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:09.796195 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5177fbbe-15f1-4b55-84dd-1f2d2a952617-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:09.797870 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:09.797843 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5177fbbe-15f1-4b55-84dd-1f2d2a952617-maas-api-xks-ca" pod "5177fbbe-15f1-4b55-84dd-1f2d2a952617" (UID: "5177fbbe-15f1-4b55-84dd-1f2d2a952617"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:45:09.797976 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:09.797936 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5177fbbe-15f1-4b55-84dd-1f2d2a952617-kube-api-access-plmwf" pod "5177fbbe-15f1-4b55-84dd-1f2d2a952617" (UID: "5177fbbe-15f1-4b55-84dd-1f2d2a952617"). InnerVolumeSpecName "kube-api-access-plmwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:09.897038 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:09.897002 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5177fbbe-15f1-4b55-84dd-1f2d2a952617-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:09.897207 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:09.897055 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plmwf\" (UniqueName: \"kubernetes.io/projected/5177fbbe-15f1-4b55-84dd-1f2d2a952617-kube-api-access-plmwf\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:10.487216 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:10.486244 5601 scope.go:122] "RemoveContainer" containerID="52b1079b203f2bb6bd427228092f1d93dd51399ba049f9bc51654769dda07637" Nov 02 17:45:10.487216 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:45:10.486690 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q_odh-ai-gateway-infra(d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" podUID="d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af" Nov 02 17:45:10.487946 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:10.487827 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m" event={"ID":"5177fbbe-15f1-4b55-84dd-1f2d2a952617","Type":"ContainerDied","Data":"353d69563b3b5a453042c9140cc96baa812f49f09eeb01b09dff4975c79948d2"} Nov 02 17:45:10.487946 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:10.487886 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="353d69563b3b5a453042c9140cc96baa812f49f09eeb01b09dff4975c79948d2" Nov 02 17:45:10.487946 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:10.487842 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m" Nov 02 17:45:10.776350 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:10.776267 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 17:45:10.776350 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:10.776326 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" podUID="e162853a-15f2-42ad-b350-bbc126c32272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 17:45:11.776286 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:11.776254 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 17:45:11.776684 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:11.776303 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" podUID="e162853a-15f2-42ad-b350-bbc126c32272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 17:45:12.783640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:12.782903 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 17:45:12.783640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:12.782962 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" podUID="e162853a-15f2-42ad-b350-bbc126c32272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 17:45:14.122643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:14.118745 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:14.450046 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:14.450012 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:14.450956 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:14.450927 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:14.451895 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:14.451876 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 17:45:16.740575 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.736531 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9"] Nov 02 17:45:16.740575 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.737359 5601 cpu_manager.go:414] "containerMap: removing container" podUID="5177fbbe-15f1-4b55-84dd-1f2d2a952617" containerName="revoke-keys" Nov 02 17:45:16.740575 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.737379 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5177fbbe-15f1-4b55-84dd-1f2d2a952617" containerName="revoke-keys" Nov 02 17:45:16.740575 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.737551 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="5177fbbe-15f1-4b55-84dd-1f2d2a952617" containerName="revoke-keys" Nov 02 17:45:16.741557 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.741536 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:16.751798 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.750608 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-77f103\"/\"default-dockercfg-8h6zz\"" Nov 02 17:45:16.751798 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.750735 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-77f103\"/\"distinct-w4-77f103-kserve-self-signed-certs\"" Nov 02 17:45:16.751798 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.750921 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-77f103\"/\"openshift-service-ca.crt\"" Nov 02 17:45:16.751798 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.750993 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-77f103\"/\"kube-root-ca.crt\"" Nov 02 17:45:16.755060 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.755033 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9"] Nov 02 17:45:16.853065 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.853025 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-tls-certs\") pod \"distinct-w4-77f103-kserve-f865fd7f9-fhjj9\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:16.853232 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.853143 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-model-cache\") pod \"distinct-w4-77f103-kserve-f865fd7f9-fhjj9\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:16.853232 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.853172 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-dshm\") pod \"distinct-w4-77f103-kserve-f865fd7f9-fhjj9\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:16.853335 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.853245 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q9tm\" (UniqueName: \"kubernetes.io/projected/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-kube-api-access-7q9tm\") pod \"distinct-w4-77f103-kserve-f865fd7f9-fhjj9\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:16.853394 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.853362 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-tmp-dir\") pod \"distinct-w4-77f103-kserve-f865fd7f9-fhjj9\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:16.853553 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.853537 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-home\") pod \"distinct-w4-77f103-kserve-f865fd7f9-fhjj9\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:16.955245 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.955201 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-home\") pod \"distinct-w4-77f103-kserve-f865fd7f9-fhjj9\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:16.955444 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.955285 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-tls-certs\") pod \"distinct-w4-77f103-kserve-f865fd7f9-fhjj9\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:16.955444 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.955340 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-model-cache\") pod \"distinct-w4-77f103-kserve-f865fd7f9-fhjj9\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:16.955444 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.955360 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-dshm\") pod \"distinct-w4-77f103-kserve-f865fd7f9-fhjj9\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:16.955444 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.955375 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q9tm\" (UniqueName: \"kubernetes.io/projected/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-kube-api-access-7q9tm\") pod \"distinct-w4-77f103-kserve-f865fd7f9-fhjj9\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:16.955444 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.955400 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-tmp-dir\") pod \"distinct-w4-77f103-kserve-f865fd7f9-fhjj9\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:16.956024 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.955850 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-model-cache\") pod \"distinct-w4-77f103-kserve-f865fd7f9-fhjj9\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:16.956024 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.955900 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-tmp-dir\") pod \"distinct-w4-77f103-kserve-f865fd7f9-fhjj9\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:16.956024 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.955969 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-home\") pod \"distinct-w4-77f103-kserve-f865fd7f9-fhjj9\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:16.957955 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.957927 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-dshm\") pod \"distinct-w4-77f103-kserve-f865fd7f9-fhjj9\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:16.958854 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.958827 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-tls-certs\") pod \"distinct-w4-77f103-kserve-f865fd7f9-fhjj9\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:16.966882 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:16.966847 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q9tm\" (UniqueName: \"kubernetes.io/projected/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-kube-api-access-7q9tm\") pod \"distinct-w4-77f103-kserve-f865fd7f9-fhjj9\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:17.061074 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:17.060989 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:17.348817 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:17.348784 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9"] Nov 02 17:45:17.349672 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:45:17.349606 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdb98d3f_fb16_46dd_8c9d_8932a8fc47ed.slice/crio-16b8c0b4f78ae234a5bbdd1b67a362b6c4eb3821af85730253f75cfb8ea93e1f WatchSource:0}: Error finding container 16b8c0b4f78ae234a5bbdd1b67a362b6c4eb3821af85730253f75cfb8ea93e1f: Status 404 returned error can't find the container with id 16b8c0b4f78ae234a5bbdd1b67a362b6c4eb3821af85730253f75cfb8ea93e1f Nov 02 17:45:17.591828 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:17.591792 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" event={"ID":"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed","Type":"ContainerStarted","Data":"b70697e3ce84ae0a370376e8afc1c8c3d96021b15e1f17fdea3cf2d3441927a4"} Nov 02 17:45:17.591986 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:17.591840 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" event={"ID":"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed","Type":"ContainerStarted","Data":"16b8c0b4f78ae234a5bbdd1b67a362b6c4eb3821af85730253f75cfb8ea93e1f"} Nov 02 17:45:17.617549 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:17.617432 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podStartSLOduration=1.617417134 podStartE2EDuration="1.617417134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:17.612687144 +0000 UTC m=+682.303741158" watchObservedRunningTime="2026-11-02 17:45:17.617417134 +0000 UTC m=+682.308471168" Nov 02 17:45:18.061791 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:18.061747 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:20.419988 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:20.419947 5601 scope.go:122] "RemoveContainer" containerID="52b1079b203f2bb6bd427228092f1d93dd51399ba049f9bc51654769dda07637" Nov 02 17:45:21.635404 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:21.635312 5601 generic.go:356] "Generic (PLEG): container finished" podID="d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af" containerID="4cabf5d6c47270a3eae85dfe105f545f6caae1da4da6196845505b0d8f097616" exitCode=60 Nov 02 17:45:21.635404 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:21.635370 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" event={"ID":"d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af","Type":"ContainerDied","Data":"4cabf5d6c47270a3eae85dfe105f545f6caae1da4da6196845505b0d8f097616"} Nov 02 17:45:21.636021 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:21.635432 5601 scope.go:122] "RemoveContainer" containerID="52b1079b203f2bb6bd427228092f1d93dd51399ba049f9bc51654769dda07637" Nov 02 17:45:21.636504 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:21.636477 5601 scope.go:122] "RemoveContainer" containerID="4cabf5d6c47270a3eae85dfe105f545f6caae1da4da6196845505b0d8f097616" Nov 02 17:45:21.636947 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:45:21.636841 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q_odh-ai-gateway-infra(d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" podUID="d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af" Nov 02 17:45:22.681127 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:22.681093 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q"] Nov 02 17:45:22.832469 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:22.832442 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" Nov 02 17:45:22.971204 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:22.971072 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af-kube-api-access-nz9rh\" (UniqueName: \"kubernetes.io/projected/d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af-kube-api-access-nz9rh\") pod \"d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af\" (UID: \"d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af\") " Nov 02 17:45:22.971369 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:22.971251 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af-maas-api-service-ca\") pod \"d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af\" (UID: \"d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af\") " Nov 02 17:45:23.009073 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:23.009007 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af-maas-api-service-ca" pod "d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af" (UID: "d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:23.013859 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:23.012801 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af-kube-api-access-nz9rh" pod "d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af" (UID: "d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af"). InnerVolumeSpecName "kube-api-access-nz9rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:23.074078 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:23.074036 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:23.074078 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:23.074076 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz9rh\" (UniqueName: \"kubernetes.io/projected/d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af-kube-api-access-nz9rh\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:23.696995 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:23.696965 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" Nov 02 17:45:23.698184 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:23.697564 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q" event={"ID":"d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af","Type":"ContainerDied","Data":"ff96cb2e8569b7c199458e61f81a6854303d9594afd9233a78cea6a503addb59"} Nov 02 17:45:23.698402 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:23.698370 5601 scope.go:122] "RemoveContainer" containerID="4cabf5d6c47270a3eae85dfe105f545f6caae1da4da6196845505b0d8f097616" Nov 02 17:45:23.760638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:23.759655 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q"] Nov 02 17:45:23.823377 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:23.823342 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894025-blx2q"] Nov 02 17:45:25.386642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:25.380697 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:25.390637 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:25.387790 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:25.400637 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:25.391681 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:25.466670 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:25.466041 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af" path="/var/lib/kubelet/pods/d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af/volumes" Nov 02 17:45:25.648832 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:25.648723 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:26.129643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:26.128710 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:26.367647 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:26.367533 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:26.406696 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:26.403832 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:26.406696 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:26.404372 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:45:26.406696 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:26.405724 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 17:45:26.406696 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:26.406065 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:45:26.410758 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:26.403775 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 02 17:45:26.421700 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:26.411067 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:45:26.435672 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:26.435268 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:26.656763 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:26.656664 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:45:27.134650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:27.115699 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:27.368642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:27.368217 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:27.372640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:27.369067 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:45:27.412689 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:27.411705 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:27.412689 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:27.412156 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 17:45:27.947637 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:27.946745 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Nov 02 17:45:28.123756 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:28.123693 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:28.124818 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:28.124734 5601 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded" Nov 02 17:45:28.339117 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:28.339019 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:45:28.459725 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:28.443883 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:28.459725 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:28.444020 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 17:45:28.590648 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:28.588180 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:45:28.620639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:28.617885 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:45:34.491333 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:45:34.491273 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:45:37.029674 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.029636 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz"] Nov 02 17:45:37.030198 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.030184 5601 cpu_manager.go:414] "containerMap: removing container" podUID="d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af" containerName="cleanup" Nov 02 17:45:37.030198 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.030199 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af" containerName="cleanup" Nov 02 17:45:37.030294 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.030211 5601 cpu_manager.go:414] "containerMap: removing container" podUID="d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af" containerName="cleanup" Nov 02 17:45:37.030294 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.030216 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af" containerName="cleanup" Nov 02 17:45:37.030294 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.030229 5601 cpu_manager.go:414] "containerMap: removing container" podUID="d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af" containerName="cleanup" Nov 02 17:45:37.030294 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.030234 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af" containerName="cleanup" Nov 02 17:45:37.030463 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.030321 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af" containerName="cleanup" Nov 02 17:45:37.030463 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.030328 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af" containerName="cleanup" Nov 02 17:45:37.030463 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.030336 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f3d06b-3a84-4f22-90e8-3fce0dcdd3af" containerName="cleanup" Nov 02 17:45:37.038760 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.038730 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.048266 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.048239 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz"] Nov 02 17:45:37.075188 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.075158 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:37.087668 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.087645 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:37.172691 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.172650 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.172873 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.172722 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6zkl\" (UniqueName: \"kubernetes.io/projected/5eb03e7c-1886-4f63-ab93-312dbe0913cf-kube-api-access-p6zkl\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.172950 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.172875 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.172950 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.172919 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.173063 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.173014 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.173109 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.173061 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.173206 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.173181 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.173269 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.173257 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.173322 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.173281 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.277456 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.277415 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.277731 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.277512 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.277731 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.277538 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.277731 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.277600 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.277731 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.277673 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6zkl\" (UniqueName: \"kubernetes.io/projected/5eb03e7c-1886-4f63-ab93-312dbe0913cf-kube-api-access-p6zkl\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.277731 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.277710 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.278054 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.277735 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.278054 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.277791 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.278054 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.277842 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.280235 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.278336 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.280235 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.279037 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.280235 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.279985 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.280469 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.280270 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.284824 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.281933 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.284824 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.283400 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.285540 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.285515 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.289463 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.289438 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.293809 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.293786 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6zkl\" (UniqueName: \"kubernetes.io/projected/5eb03e7c-1886-4f63-ab93-312dbe0913cf-kube-api-access-p6zkl\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.366518 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.366487 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:37.658272 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.658241 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz"] Nov 02 17:45:37.658416 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:45:37.658398 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb03e7c_1886_4f63_ab93_312dbe0913cf.slice/crio-5d2225ba69c2aa1b432a1110a183da46346e00981c23c39b05ab58b9a95265f0 WatchSource:0}: Error finding container 5d2225ba69c2aa1b432a1110a183da46346e00981c23c39b05ab58b9a95265f0: Status 404 returned error can't find the container with id 5d2225ba69c2aa1b432a1110a183da46346e00981c23c39b05ab58b9a95265f0 Nov 02 17:45:37.661351 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.661315 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:37.661483 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.661396 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:37.661483 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:37.661426 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:38.366342 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:38.366303 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" event={"ID":"5eb03e7c-1886-4f63-ab93-312dbe0913cf","Type":"ContainerStarted","Data":"7d0018b44e7ff07a2071ebaf1f8c53d85d36a6baf02a4be71cfbb9430dc80ecb"} Nov 02 17:45:38.366777 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:38.366363 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" event={"ID":"5eb03e7c-1886-4f63-ab93-312dbe0913cf","Type":"ContainerStarted","Data":"5d2225ba69c2aa1b432a1110a183da46346e00981c23c39b05ab58b9a95265f0"} Nov 02 17:45:38.366777 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:38.366632 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:38.368394 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:38.368368 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 17:45:38.368510 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:38.368419 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" podUID="5eb03e7c-1886-4f63-ab93-312dbe0913cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 17:45:38.403006 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:38.402953 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" podStartSLOduration=1.402941134 podStartE2EDuration="1.402941134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:38.400146441 +0000 UTC m=+703.091200454" watchObservedRunningTime="2026-11-02 17:45:38.402941134 +0000 UTC m=+703.093995141" Nov 02 17:45:39.372826 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:39.372777 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 17:45:39.373337 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:39.372848 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" podUID="5eb03e7c-1886-4f63-ab93-312dbe0913cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 17:45:40.367123 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:40.367084 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 17:45:40.367354 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:40.367139 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" podUID="5eb03e7c-1886-4f63-ab93-312dbe0913cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 17:45:41.372327 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:41.372292 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 17:45:41.372936 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:41.372897 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" podUID="5eb03e7c-1886-4f63-ab93-312dbe0913cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 17:45:42.347769 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:42.347713 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:42.368048 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:42.368011 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 17:45:42.368366 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:42.368333 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" podUID="5eb03e7c-1886-4f63-ab93-312dbe0913cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 17:45:42.453815 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:42.453756 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:42.483646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:42.477987 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:42.595182 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:42.595137 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:42.640889 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:42.640769 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 02 17:45:42.641758 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:42.641727 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:43.354641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:43.351655 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:43.354641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:43.352553 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:45:43.369334 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:43.368971 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 17:45:43.369334 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:43.369025 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" podUID="5eb03e7c-1886-4f63-ab93-312dbe0913cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 17:45:43.629159 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:43.601919 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:43.629159 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:43.628088 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:43.657646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:43.648994 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:43.657646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:43.649820 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:43.895174 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:43.895083 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:45:43.920845 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:43.920801 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:45:43.921323 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:43.921304 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:45:43.921576 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:43.921560 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:45:44.149695 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:44.126910 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:44.249533 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:44.249499 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:45:44.388813 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:44.387388 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 17:45:44.388813 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:44.387457 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" podUID="5eb03e7c-1886-4f63-ab93-312dbe0913cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 17:45:44.400795 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:44.400703 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:44.544771 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:44.544715 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:44.607650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:44.606823 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:44.640639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:44.630754 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:44.640639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:44.631274 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:45:44.687879 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:44.687832 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:44.731277 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:44.731238 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:45.134711 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:45.128919 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:45.134711 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:45.130020 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:45.134711 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:45.130364 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:45.283787 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:45.283748 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 17:45:45.374080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:45.374042 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 17:45:45.374330 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:45.374303 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" podUID="5eb03e7c-1886-4f63-ab93-312dbe0913cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 17:45:45.549751 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:45.549703 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:45.569677 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:45.550683 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:45:45.618056 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:45.618012 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:45.644001 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:45.643963 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 02 17:45:45.689936 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:45.688851 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:45.750290 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:45.746909 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 17:45:45.755140 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:45.753792 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:45:45.962552 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:45.962526 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:45:46.134216 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:46.134170 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:46.292362 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:46.291509 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:45:46.331269 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:46.327661 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 17:45:46.331269 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:46.328240 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:45:46.367542 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:46.367508 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 17:45:46.367826 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:46.367794 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" podUID="5eb03e7c-1886-4f63-ab93-312dbe0913cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 17:45:46.407647 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:46.407360 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:45:46.483132 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:46.483103 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:45:46.493009 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:46.492988 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:45:46.496778 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:46.496757 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:45:46.505088 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:46.505067 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:45:47.753576 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:47.753537 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:47.754073 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:47.753936 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:47.755345 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:47.755319 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:45:52.233399 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.233361 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94"] Nov 02 17:45:52.238078 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.238055 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.247932 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.247880 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-77f103-openshift-default-dockercfg-v9fvh\"" Nov 02 17:45:52.256639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.252205 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94"] Nov 02 17:45:52.290536 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.290410 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.290895 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.290871 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.291031 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.291015 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.291886 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.291862 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.292056 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.292037 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.292190 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.292173 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171d0002-0cae-4bb8-a398-9ba0da9f393f-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.292453 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.292333 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckfht\" (UniqueName: \"kubernetes.io/projected/171d0002-0cae-4bb8-a398-9ba0da9f393f-kube-api-access-ckfht\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.292638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.292599 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.292808 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.292787 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.399433 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.399366 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.399644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.399469 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.399644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.399504 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.399644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.399530 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.399644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.399604 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.399935 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.399657 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.399935 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.399682 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171d0002-0cae-4bb8-a398-9ba0da9f393f-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.399935 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.399718 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckfht\" (UniqueName: \"kubernetes.io/projected/171d0002-0cae-4bb8-a398-9ba0da9f393f-kube-api-access-ckfht\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.399935 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.399765 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.399935 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.399907 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.400204 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.399950 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.400204 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.400071 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.400884 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.400854 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171d0002-0cae-4bb8-a398-9ba0da9f393f-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.401012 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.400893 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.402944 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.402901 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.404860 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.404835 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.408479 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.408446 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.408865 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.408840 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckfht\" (UniqueName: \"kubernetes.io/projected/171d0002-0cae-4bb8-a398-9ba0da9f393f-kube-api-access-ckfht\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:52.594891 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:52.594805 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:54.239701 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:45:54.229976 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171d0002_0cae_4bb8_a398_9ba0da9f393f.slice/crio-fb7101df07518f54d722aa371a6b867b2cf0fc0a306e0088eb8b5298405865b0 WatchSource:0}: Error finding container fb7101df07518f54d722aa371a6b867b2cf0fc0a306e0088eb8b5298405865b0: Status 404 returned error can't find the container with id fb7101df07518f54d722aa371a6b867b2cf0fc0a306e0088eb8b5298405865b0 Nov 02 17:45:54.256865 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:54.256808 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:54.257024 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:54.256912 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:54.257024 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:54.256958 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:45:54.282299 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:54.257437 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94"] Nov 02 17:45:54.387149 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:54.386698 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:54.533777 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:54.533723 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:54.548571 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:54.540734 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:54.664333 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:54.662501 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:54.664333 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:54.662919 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 17:45:54.883751 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:54.882610 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" event={"ID":"171d0002-0cae-4bb8-a398-9ba0da9f393f","Type":"ContainerStarted","Data":"87fbcd32e41cedf2ac27888124ecfd468ba798670c7474ecd87de360104224a3"} Nov 02 17:45:54.883751 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:54.882697 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" event={"ID":"171d0002-0cae-4bb8-a398-9ba0da9f393f","Type":"ContainerStarted","Data":"fb7101df07518f54d722aa371a6b867b2cf0fc0a306e0088eb8b5298405865b0"} Nov 02 17:45:54.935849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:54.935784 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" podStartSLOduration=2.935763343 podStartE2EDuration="2.935763343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:54.920277143 +0000 UTC m=+719.611331157" watchObservedRunningTime="2026-11-02 17:45:54.935763343 +0000 UTC m=+719.626817356" Nov 02 17:45:55.596704 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:55.596670 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:45:55.598230 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:55.598202 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 17:45:55.598284 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:55.598258 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" podUID="171d0002-0cae-4bb8-a398-9ba0da9f393f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 17:45:56.597218 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:56.597181 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 17:45:56.597643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:56.597246 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" podUID="171d0002-0cae-4bb8-a398-9ba0da9f393f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 17:45:57.280464 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.280425 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s"] Nov 02 17:45:57.287896 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.287867 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.299356 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.299330 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-77f103\"/\"distinct-2-w4-77f103-kserve-self-signed-certs\"" Nov 02 17:45:57.302517 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.302478 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s"] Nov 02 17:45:57.402374 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.402342 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b595c7d-1c54-412a-9035-79633dcd1f24-tls-certs\") pod \"distinct-2-w4-77f103-kserve-68f97f49fb-6478s\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.402656 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.402638 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-home\") pod \"distinct-2-w4-77f103-kserve-68f97f49fb-6478s\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.402842 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.402822 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-dshm\") pod \"distinct-2-w4-77f103-kserve-68f97f49fb-6478s\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.403023 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.403012 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-tmp-dir\") pod \"distinct-2-w4-77f103-kserve-68f97f49fb-6478s\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.403117 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.403105 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-model-cache\") pod \"distinct-2-w4-77f103-kserve-68f97f49fb-6478s\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.403200 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.403189 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zrjf\" (UniqueName: \"kubernetes.io/projected/4b595c7d-1c54-412a-9035-79633dcd1f24-kube-api-access-6zrjf\") pod \"distinct-2-w4-77f103-kserve-68f97f49fb-6478s\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.509489 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.509446 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-tmp-dir\") pod \"distinct-2-w4-77f103-kserve-68f97f49fb-6478s\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.510256 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.510234 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-model-cache\") pod \"distinct-2-w4-77f103-kserve-68f97f49fb-6478s\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.510732 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.510711 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zrjf\" (UniqueName: \"kubernetes.io/projected/4b595c7d-1c54-412a-9035-79633dcd1f24-kube-api-access-6zrjf\") pod \"distinct-2-w4-77f103-kserve-68f97f49fb-6478s\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.511326 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.511305 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b595c7d-1c54-412a-9035-79633dcd1f24-tls-certs\") pod \"distinct-2-w4-77f103-kserve-68f97f49fb-6478s\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.511931 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.511914 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-home\") pod \"distinct-2-w4-77f103-kserve-68f97f49fb-6478s\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.512402 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.510650 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-model-cache\") pod \"distinct-2-w4-77f103-kserve-68f97f49fb-6478s\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.512535 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.510163 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-tmp-dir\") pod \"distinct-2-w4-77f103-kserve-68f97f49fb-6478s\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.512651 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.512298 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-home\") pod \"distinct-2-w4-77f103-kserve-68f97f49fb-6478s\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.512767 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.512753 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-dshm\") pod \"distinct-2-w4-77f103-kserve-68f97f49fb-6478s\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.526711 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.526682 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zrjf\" (UniqueName: \"kubernetes.io/projected/4b595c7d-1c54-412a-9035-79633dcd1f24-kube-api-access-6zrjf\") pod \"distinct-2-w4-77f103-kserve-68f97f49fb-6478s\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.543354 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.543276 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-dshm\") pod \"distinct-2-w4-77f103-kserve-68f97f49fb-6478s\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.544183 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.544154 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b595c7d-1c54-412a-9035-79633dcd1f24-tls-certs\") pod \"distinct-2-w4-77f103-kserve-68f97f49fb-6478s\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.600137 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.600092 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 17:45:57.600775 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.600728 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" podUID="171d0002-0cae-4bb8-a398-9ba0da9f393f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 17:45:57.612567 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.612529 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:45:57.971656 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:57.971605 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s"] Nov 02 17:45:57.972598 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:45:57.972569 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b595c7d_1c54_412a_9035_79633dcd1f24.slice/crio-8b55d4baea151a59fa780426a6da3db91caff5fe65f435b51296747a4bfb3bdd WatchSource:0}: Error finding container 8b55d4baea151a59fa780426a6da3db91caff5fe65f435b51296747a4bfb3bdd: Status 404 returned error can't find the container with id 8b55d4baea151a59fa780426a6da3db91caff5fe65f435b51296747a4bfb3bdd Nov 02 17:45:58.598103 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:58.598053 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 17:45:58.598443 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:58.598406 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" podUID="171d0002-0cae-4bb8-a398-9ba0da9f393f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 17:45:58.909901 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:58.909818 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" event={"ID":"4b595c7d-1c54-412a-9035-79633dcd1f24","Type":"ContainerStarted","Data":"f29c3a86d5ed3b1914566cc097be0ca9b52d424bd7b408ddd4b8c5172fbf81fb"} Nov 02 17:45:58.909901 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:58.909860 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" event={"ID":"4b595c7d-1c54-412a-9035-79633dcd1f24","Type":"ContainerStarted","Data":"8b55d4baea151a59fa780426a6da3db91caff5fe65f435b51296747a4bfb3bdd"} Nov 02 17:45:58.931825 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:58.931766 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podStartSLOduration=1.931752441 podStartE2EDuration="1.931752441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:58.930015514 +0000 UTC m=+723.621069526" watchObservedRunningTime="2026-11-02 17:45:58.931752441 +0000 UTC m=+723.622806453" Nov 02 17:45:59.597787 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:59.597749 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 17:45:59.597957 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:59.597818 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" podUID="171d0002-0cae-4bb8-a398-9ba0da9f393f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 17:45:59.614007 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:45:59.613981 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:46:00.597289 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:00.597257 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 17:46:00.597755 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:00.597713 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" podUID="171d0002-0cae-4bb8-a398-9ba0da9f393f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 17:46:01.614639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:01.601061 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 17:46:01.614639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:01.601124 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" podUID="171d0002-0cae-4bb8-a398-9ba0da9f393f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 17:46:01.827827 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:01.827766 5601 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:46:01.828124 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:01.828097 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" podUID="db8e57a9-62e6-4229-921d-f5a40cde2a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:03.579384 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:03.579342 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:46:03.582705 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:03.582665 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:46:03.586197 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:03.586170 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 17:46:04.488382 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:46:04.488340 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:46:07.613585 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:07.613498 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:46:07.626682 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:07.626654 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:46:08.014699 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:08.014669 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:46:12.576786 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:12.576732 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:12.621749 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:12.621700 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:12.962886 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:12.962819 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:13.007766 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:13.007714 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:13.128642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:13.127931 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:13.307954 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:13.306706 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:13.480642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:13.480382 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:13.579804 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:13.579698 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:13.581102 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:13.581076 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:46:13.642120 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:13.642078 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:13.643196 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:13.643157 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:46:13.646297 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:13.646269 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:13.775640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:13.737780 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 02 17:46:13.859697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:13.858002 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:14.007642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:13.973018 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:14.007642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:13.973799 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:46:14.011826 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.011749 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:14.065668 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.012081 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:14.065668 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.042283 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:46:14.141805 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.139009 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:14.141805 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.139741 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:46:14.311834 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.311786 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:14.420494 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.333538 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:46:14.423064 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.421154 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:14.438741 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.438387 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:46:14.442761 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.442739 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:46:14.508641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.503968 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 17:46:14.508641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.504734 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:46:14.617743 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.586082 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:14.703494 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.660854 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:14.728644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.660892 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 17:46:14.728644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.728465 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:46:14.859485 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.859444 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 02 17:46:14.861004 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.860979 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:46:14.885302 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.885274 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:46:14.921309 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.921281 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:46:14.942132 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:14.942107 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:46:15.251661 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:15.248222 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:46:15.335743 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:15.335695 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 02 17:46:15.425412 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:15.425382 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:46:15.447567 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:15.446857 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:15.454054 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:15.453689 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:15.543705 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:15.541305 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:46:15.670681 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:15.669954 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:46:28.436652 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:28.434721 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:28.453651 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:28.452904 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 17:46:28.529809 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:28.529754 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:28.636242 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:28.636184 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded" Nov 02 17:46:28.738903 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:28.738768 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:28.905551 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:28.904327 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:29.044928 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:29.044797 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:29.254721 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:29.254676 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:29.454688 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:29.451005 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:29.454688 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:29.451458 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:29.455295 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:29.454724 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:29.533482 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:29.533446 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:46:29.533855 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:29.533836 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:46:29.564641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:29.561138 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:29.564641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:29.562004 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:46:29.779697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:29.763040 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:29.805643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:29.780714 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:46:29.914657 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:29.905651 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:29.914657 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:29.906121 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:46:29.962652 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:29.957122 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 02 17:46:30.050698 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:30.050070 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:30.050698 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:30.050573 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:46:30.204041 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:30.204004 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:46:30.204380 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:30.204293 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" podUID="5c1fcd8f-3977-4d4b-826c-38150e08228c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:30.452972 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:30.452925 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 17:46:30.471252 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:30.469669 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:30.498528 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:30.498486 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:46:30.498841 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:30.498809 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" podUID="e162853a-15f2-42ad-b350-bbc126c32272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:30.612651 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:30.604059 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 17:46:30.833137 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:30.833036 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:30.960643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:30.952656 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:30.994695 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:30.982200 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 02 17:46:30.998927 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:30.997801 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:46:31.065987 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:31.065945 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:31.263975 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:31.263879 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:31.316821 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:31.316773 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:31.370642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:31.362256 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:46:31.370642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:31.362321 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" podUID="398ed64f-2082-405d-a5c3-3628b0fdd21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:31.513642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:31.510503 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 17:46:31.531118 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:31.528787 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:31.531118 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:31.529148 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 17:46:31.615691 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:31.615648 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:31.810928 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:31.810833 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-jv8f4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:46:31.811194 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:31.811160 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" podUID="c32fa6f3-9665-4b3b-b093-22435190829b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:31.864769 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:31.864719 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:31.991770 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:31.991719 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:32.079754 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:32.079657 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-wtks5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:46:32.080405 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:32.079981 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" podUID="63f70366-70f1-475a-8e45-eb6da6b4828e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:32.080968 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:32.080936 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 02 17:46:32.131039 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:32.130993 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:32.149817 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:32.149775 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Nov 02 17:46:32.281641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:32.265878 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:32.281641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:32.266745 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:46:32.356217 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:32.355774 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h95t4" podUID="0622e414-7cf5-47f3-b8d1-1b79d6e73750" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.25:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:32.417675 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:32.417607 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:46:32.450932 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:32.450878 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" podUID="ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:32.469964 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:32.468918 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:46:32.517636 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:32.512316 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:32.517636 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:32.513061 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:46:32.539532 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:32.534369 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:32.539532 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:32.534732 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 17:46:32.572501 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:32.571786 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:32.755040 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:32.754951 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h95t4" podUID="0622e414-7cf5-47f3-b8d1-1b79d6e73750" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.25:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:32.773038 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:32.773004 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:46:32.784499 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:32.784461 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 17:46:33.009693 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:33.008786 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-mdxxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:46:33.009693 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:33.008900 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" podUID="2f57ca2f-9916-45aa-9523-65e8fd925f15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:33.128364 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:33.128320 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:33.130295 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:33.130258 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:46:33.266963 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:33.266055 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 17:46:33.304281 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:33.300768 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:33.393231 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:33.383683 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:46:33.413644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:33.410944 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 17:46:33.588278 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:33.588191 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:46:33.608127 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:33.608095 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:46:33.609472 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:33.609446 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:46:33.625985 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:33.625959 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:46:34.485085 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:46:34.485049 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:46:35.824968 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:35.824933 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn"] Nov 02 17:46:35.830828 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:35.830760 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:35.841543 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:35.841473 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn"] Nov 02 17:46:35.842278 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:35.842249 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-da5f07e6-openshift-default-dockercfg-hpwn4\"" Nov 02 17:46:35.967453 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:35.967397 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-istio-envoy\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:35.967885 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:35.967858 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-credential-socket\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:35.968091 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:35.968059 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4q4c\" (UniqueName: \"kubernetes.io/projected/3f62a981-07b4-4450-9013-959356a33a36-kube-api-access-l4q4c\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:35.968272 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:35.968254 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f62a981-07b4-4450-9013-959356a33a36-istiod-ca-cert\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:35.968417 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:35.968403 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f62a981-07b4-4450-9013-959356a33a36-istio-podinfo\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:35.968675 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:35.968660 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-workload-certs\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:35.968780 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:35.968765 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-istio-data\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:35.968901 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:35.968889 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-workload-socket\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:35.968979 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:35.968967 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f62a981-07b4-4450-9013-959356a33a36-istio-token\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.070540 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.070500 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-credential-socket\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.071919 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.071896 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4q4c\" (UniqueName: \"kubernetes.io/projected/3f62a981-07b4-4450-9013-959356a33a36-kube-api-access-l4q4c\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.072439 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.072417 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f62a981-07b4-4450-9013-959356a33a36-istiod-ca-cert\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.073289 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.073267 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f62a981-07b4-4450-9013-959356a33a36-istio-podinfo\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.073596 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.073575 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-workload-certs\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.076002 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.075973 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-istio-data\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.076100 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.076076 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-workload-socket\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.076170 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.076109 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f62a981-07b4-4450-9013-959356a33a36-istio-token\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.076242 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.076222 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-istio-envoy\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.076301 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.075357 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-workload-certs\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.076367 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.073223 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f62a981-07b4-4450-9013-959356a33a36-istiod-ca-cert\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.076436 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.071824 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-credential-socket\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.076665 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.076577 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-workload-socket\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.076792 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.076771 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-istio-data\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.081351 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.081289 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-istio-envoy\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.081707 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.081662 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f62a981-07b4-4450-9013-959356a33a36-istio-podinfo\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.082462 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.082435 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4q4c\" (UniqueName: \"kubernetes.io/projected/3f62a981-07b4-4450-9013-959356a33a36-kube-api-access-l4q4c\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.084228 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.084199 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f62a981-07b4-4450-9013-959356a33a36-istio-token\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.152232 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.152167 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:36.444480 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.444443 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn"] Nov 02 17:46:36.445732 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:46:36.445695 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f62a981_07b4_4450_9013_959356a33a36.slice/crio-4c31802a1680a46a73345036242e30c50665c9bfd841efecaf16121bc0b761ba WatchSource:0}: Error finding container 4c31802a1680a46a73345036242e30c50665c9bfd841efecaf16121bc0b761ba: Status 404 returned error can't find the container with id 4c31802a1680a46a73345036242e30c50665c9bfd841efecaf16121bc0b761ba Nov 02 17:46:36.448885 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.448844 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:46:36.448979 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.448930 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:46:36.449023 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:36.448973 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:46:37.436785 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:37.436749 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" event={"ID":"3f62a981-07b4-4450-9013-959356a33a36","Type":"ContainerStarted","Data":"3c084d3b72fd58de7e76b9b67193e713b508ba8493d39cc76bc9329cedbf059f"} Nov 02 17:46:37.437199 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:37.436794 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" event={"ID":"3f62a981-07b4-4450-9013-959356a33a36","Type":"ContainerStarted","Data":"4c31802a1680a46a73345036242e30c50665c9bfd841efecaf16121bc0b761ba"} Nov 02 17:46:37.478062 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:37.477927 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" podStartSLOduration=2.477909038 podStartE2EDuration="2.477909038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:37.477737303 +0000 UTC m=+762.168791313" watchObservedRunningTime="2026-11-02 17:46:37.477909038 +0000 UTC m=+762.168963050" Nov 02 17:46:38.152687 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:38.152652 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:38.158086 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:38.158062 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:38.443998 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:38.443968 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:38.447100 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:38.445444 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:46:41.906185 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:41.906151 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh"] Nov 02 17:46:41.909835 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:41.909814 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" Nov 02 17:46:41.913957 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:41.913923 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2893d1-serving-cert\"" Nov 02 17:46:41.921921 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:41.921526 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh"] Nov 02 17:46:42.060427 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:42.060383 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbt2h\" (UniqueName: \"kubernetes.io/projected/aa7f1f3d-5879-4234-82f0-02f0100f199f-kube-api-access-kbt2h\") pod \"maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh\" (UID: \"aa7f1f3d-5879-4234-82f0-02f0100f199f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" Nov 02 17:46:42.060606 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:42.060573 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa7f1f3d-5879-4234-82f0-02f0100f199f-metrics-certs\") pod \"maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh\" (UID: \"aa7f1f3d-5879-4234-82f0-02f0100f199f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" Nov 02 17:46:42.060712 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:42.060642 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa7f1f3d-5879-4234-82f0-02f0100f199f-maas-api-tls\") pod \"maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh\" (UID: \"aa7f1f3d-5879-4234-82f0-02f0100f199f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" Nov 02 17:46:42.060791 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:42.060768 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7f1f3d-5879-4234-82f0-02f0100f199f-openshift-service-ca\") pod \"maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh\" (UID: \"aa7f1f3d-5879-4234-82f0-02f0100f199f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" Nov 02 17:46:42.162538 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:42.162450 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa7f1f3d-5879-4234-82f0-02f0100f199f-metrics-certs\") pod \"maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh\" (UID: \"aa7f1f3d-5879-4234-82f0-02f0100f199f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" Nov 02 17:46:42.162538 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:42.162499 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa7f1f3d-5879-4234-82f0-02f0100f199f-maas-api-tls\") pod \"maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh\" (UID: \"aa7f1f3d-5879-4234-82f0-02f0100f199f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" Nov 02 17:46:42.162755 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:42.162549 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7f1f3d-5879-4234-82f0-02f0100f199f-openshift-service-ca\") pod \"maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh\" (UID: \"aa7f1f3d-5879-4234-82f0-02f0100f199f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" Nov 02 17:46:42.162755 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:42.162685 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbt2h\" (UniqueName: \"kubernetes.io/projected/aa7f1f3d-5879-4234-82f0-02f0100f199f-kube-api-access-kbt2h\") pod \"maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh\" (UID: \"aa7f1f3d-5879-4234-82f0-02f0100f199f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" Nov 02 17:46:42.163244 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:42.163219 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7f1f3d-5879-4234-82f0-02f0100f199f-openshift-service-ca\") pod \"maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh\" (UID: \"aa7f1f3d-5879-4234-82f0-02f0100f199f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" Nov 02 17:46:42.165245 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:42.165221 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa7f1f3d-5879-4234-82f0-02f0100f199f-maas-api-tls\") pod \"maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh\" (UID: \"aa7f1f3d-5879-4234-82f0-02f0100f199f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" Nov 02 17:46:42.165325 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:42.165241 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa7f1f3d-5879-4234-82f0-02f0100f199f-metrics-certs\") pod \"maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh\" (UID: \"aa7f1f3d-5879-4234-82f0-02f0100f199f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" Nov 02 17:46:42.172867 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:42.172845 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbt2h\" (UniqueName: \"kubernetes.io/projected/aa7f1f3d-5879-4234-82f0-02f0100f199f-kube-api-access-kbt2h\") pod \"maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh\" (UID: \"aa7f1f3d-5879-4234-82f0-02f0100f199f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" Nov 02 17:46:42.228840 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:42.228803 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" Nov 02 17:46:42.575046 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:42.571501 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh"] Nov 02 17:46:42.575046 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:42.572633 5601 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:46:43.526692 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:43.526654 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" event={"ID":"aa7f1f3d-5879-4234-82f0-02f0100f199f","Type":"ContainerStarted","Data":"6dd7ab510bc4f9ee510eeb90fcd19130d5596b9ad0db1a96921781db800c7c54"} Nov 02 17:46:43.527060 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:43.526734 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" event={"ID":"aa7f1f3d-5879-4234-82f0-02f0100f199f","Type":"ContainerStarted","Data":"166ecce34b806cd7d93e51c6e6bb90766d355a9a5838c728718036d469d2f8db"} Nov 02 17:46:43.527060 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:43.526875 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" Nov 02 17:46:43.548807 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:43.548758 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" podStartSLOduration=2.07813152 podStartE2EDuration="2.548741816s" totalImagesPullingTime="470.610296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:43.548225282 +0000 UTC m=+768.239279294" watchObservedRunningTime="2026-11-02 17:46:43.548741816 +0000 UTC m=+768.239795828" Nov 02 17:46:49.539993 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:49.539964 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" Nov 02 17:46:52.372377 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.372342 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn"] Nov 02 17:46:52.376546 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.376522 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.391793 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.391761 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn"] Nov 02 17:46:52.544095 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.544053 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.544481 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.544460 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.544689 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.544664 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.544837 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.544792 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.544998 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.544840 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.544998 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.544936 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.545127 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.545005 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.545127 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.545028 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.545127 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.545115 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngblj\" (UniqueName: \"kubernetes.io/projected/0077aa8d-4c02-40d7-ac71-beb13830dcf9-kube-api-access-ngblj\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.646249 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.646148 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngblj\" (UniqueName: \"kubernetes.io/projected/0077aa8d-4c02-40d7-ac71-beb13830dcf9-kube-api-access-ngblj\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.646428 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.646256 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.646428 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.646279 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.646428 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.646402 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.646611 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.646498 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.646611 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.646550 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.646743 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.646643 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.646743 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.646704 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.646743 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.646735 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.646901 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.646743 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.646901 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.646851 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.647008 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.646988 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.647046 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.647031 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.647268 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.647246 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.648610 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.648588 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.649088 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.649072 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.655295 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.655273 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.655452 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.655432 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngblj\" (UniqueName: \"kubernetes.io/projected/0077aa8d-4c02-40d7-ac71-beb13830dcf9-kube-api-access-ngblj\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.698505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.698479 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:52.999037 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:46:52.998991 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0077aa8d_4c02_40d7_ac71_beb13830dcf9.slice/crio-a16455c8b47a0abb4c8c39b89f231e8c3591a57b80dcf7e059d5d04ef7dde28f WatchSource:0}: Error finding container a16455c8b47a0abb4c8c39b89f231e8c3591a57b80dcf7e059d5d04ef7dde28f: Status 404 returned error can't find the container with id a16455c8b47a0abb4c8c39b89f231e8c3591a57b80dcf7e059d5d04ef7dde28f Nov 02 17:46:52.999177 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:52.999106 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn"] Nov 02 17:46:53.005382 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:53.002568 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:46:53.005382 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:53.002681 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:46:53.005382 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:53.002756 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:46:53.599415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:53.599378 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" event={"ID":"0077aa8d-4c02-40d7-ac71-beb13830dcf9","Type":"ContainerStarted","Data":"bf6373fae3ef708428abe9a5b39e5b8beb0c501ebb2cc0dcb4fe3a3962790e12"} Nov 02 17:46:53.599875 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:53.599435 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" event={"ID":"0077aa8d-4c02-40d7-ac71-beb13830dcf9","Type":"ContainerStarted","Data":"a16455c8b47a0abb4c8c39b89f231e8c3591a57b80dcf7e059d5d04ef7dde28f"} Nov 02 17:46:53.619518 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:53.619468 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" podStartSLOduration=1.619454099 podStartE2EDuration="1.619454099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:53.618258518 +0000 UTC m=+778.309312530" watchObservedRunningTime="2026-11-02 17:46:53.619454099 +0000 UTC m=+778.310508107" Nov 02 17:46:53.699441 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:53.699405 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:46:54.700885 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:54.700509 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 17:46:54.700885 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:54.700576 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" podUID="0077aa8d-4c02-40d7-ac71-beb13830dcf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 17:46:55.673572 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:55.673539 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt"] Nov 02 17:46:55.677857 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:55.677837 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt" Nov 02 17:46:55.684315 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:55.684173 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-58970f\"" Nov 02 17:46:55.684315 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:55.684287 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-58970f-dockercfg-4ts5x\"" Nov 02 17:46:55.688485 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:55.688462 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt"] Nov 02 17:46:55.699720 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:55.699692 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 17:46:55.699824 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:55.699759 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" podUID="0077aa8d-4c02-40d7-ac71-beb13830dcf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 17:46:55.702509 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:55.702477 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6cn8\" (UniqueName: \"kubernetes.io/projected/c470dc21-8297-4937-9589-5cf3a292741b-kube-api-access-l6cn8\") pod \"payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt\" (UID: \"c470dc21-8297-4937-9589-5cf3a292741b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt" Nov 02 17:46:55.702926 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:55.702552 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c470dc21-8297-4937-9589-5cf3a292741b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt\" (UID: \"c470dc21-8297-4937-9589-5cf3a292741b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt" Nov 02 17:46:55.813172 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:55.803534 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6cn8\" (UniqueName: \"kubernetes.io/projected/c470dc21-8297-4937-9589-5cf3a292741b-kube-api-access-l6cn8\") pod \"payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt\" (UID: \"c470dc21-8297-4937-9589-5cf3a292741b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt" Nov 02 17:46:55.813172 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:55.803602 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c470dc21-8297-4937-9589-5cf3a292741b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt\" (UID: \"c470dc21-8297-4937-9589-5cf3a292741b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt" Nov 02 17:46:55.814266 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:55.813915 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c470dc21-8297-4937-9589-5cf3a292741b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt\" (UID: \"c470dc21-8297-4937-9589-5cf3a292741b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt" Nov 02 17:46:55.885017 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:55.883417 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6cn8\" (UniqueName: \"kubernetes.io/projected/c470dc21-8297-4937-9589-5cf3a292741b-kube-api-access-l6cn8\") pod \"payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt\" (UID: \"c470dc21-8297-4937-9589-5cf3a292741b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt" Nov 02 17:46:55.993005 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:55.992914 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt" Nov 02 17:46:56.526448 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:56.526416 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt"] Nov 02 17:46:56.527921 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:46:56.527892 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc470dc21_8297_4937_9589_5cf3a292741b.slice/crio-6478fe2adb2fcc52a63c04dec514c8ed439737f992eb8845990c81a53d12db89 WatchSource:0}: Error finding container 6478fe2adb2fcc52a63c04dec514c8ed439737f992eb8845990c81a53d12db89: Status 404 returned error can't find the container with id 6478fe2adb2fcc52a63c04dec514c8ed439737f992eb8845990c81a53d12db89 Nov 02 17:46:56.623222 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:56.623173 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt" event={"ID":"c470dc21-8297-4937-9589-5cf3a292741b","Type":"ContainerStarted","Data":"6478fe2adb2fcc52a63c04dec514c8ed439737f992eb8845990c81a53d12db89"} Nov 02 17:46:56.699049 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:56.699022 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 17:46:56.699190 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:56.699077 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" podUID="0077aa8d-4c02-40d7-ac71-beb13830dcf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 17:46:57.631002 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:57.630965 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt" event={"ID":"c470dc21-8297-4937-9589-5cf3a292741b","Type":"ContainerStarted","Data":"c13ce7b5a9efd8ab1ccc1bc4dc72827db6825db88629ac45f103d94691508e30"} Nov 02 17:46:57.631417 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:57.631049 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt" Nov 02 17:46:57.660537 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:57.660477 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt" podStartSLOduration=2.660461334 podStartE2EDuration="2.660461334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:57.659112612 +0000 UTC m=+782.350166628" watchObservedRunningTime="2026-11-02 17:46:57.660461334 +0000 UTC m=+782.351515349" Nov 02 17:46:57.699111 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:57.699077 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 17:46:57.699274 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:57.699137 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" podUID="0077aa8d-4c02-40d7-ac71-beb13830dcf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 17:46:57.997123 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:57.997084 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw"] Nov 02 17:46:58.002937 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.002905 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.015037 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.015011 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-935086-openshift-default-dockercfg-xd4zg\"" Nov 02 17:46:58.016772 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.016744 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw"] Nov 02 17:46:58.073608 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.073450 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istiod-ca-cert\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.073608 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.073531 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-data\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.073608 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.073599 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-token\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.074005 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.073776 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-workload-certs\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.074005 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.073868 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-credential-socket\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.074005 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.073930 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-workload-socket\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.074005 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.073948 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7xjm\" (UniqueName: \"kubernetes.io/projected/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-kube-api-access-j7xjm\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.074005 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.073967 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-envoy\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.074005 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.073988 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-podinfo\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.186221 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.176247 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-workload-certs\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.186221 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.176361 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-credential-socket\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.186221 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.176434 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-workload-socket\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.186221 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.176463 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7xjm\" (UniqueName: \"kubernetes.io/projected/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-kube-api-access-j7xjm\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.186221 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.176492 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-envoy\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.186221 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.176525 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-podinfo\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.186221 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.176663 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istiod-ca-cert\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.186221 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.176720 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-data\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.186221 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.176744 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-token\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.186221 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.177532 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-workload-certs\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.186221 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.177804 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-credential-socket\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.186221 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.178073 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-workload-socket\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.186221 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.179314 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istiod-ca-cert\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.203643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.199071 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-data\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.217999 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.217924 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-envoy\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.225804 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.222032 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7xjm\" (UniqueName: \"kubernetes.io/projected/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-kube-api-access-j7xjm\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.226351 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.226326 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-token\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.239639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.231388 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-podinfo\") pod \"e2e-api-b-935086-openshift-default-8497785c69-j2mzw\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.355973 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.355878 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:46:58.683262 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.683229 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw"] Nov 02 17:46:58.686931 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:46:58.686904 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe64c3d_8ff3_400f_8dc9_d7c8edcdbb18.slice/crio-a4fe2af33fe9c3a1793ad00b08c69bd68acd90e799855c81f617a6fe2b5811d8 WatchSource:0}: Error finding container a4fe2af33fe9c3a1793ad00b08c69bd68acd90e799855c81f617a6fe2b5811d8: Status 404 returned error can't find the container with id a4fe2af33fe9c3a1793ad00b08c69bd68acd90e799855c81f617a6fe2b5811d8 Nov 02 17:46:58.689916 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.689884 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:46:58.690022 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.689963 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:46:58.690022 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.690009 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:46:58.699473 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.699446 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 17:46:58.699574 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:58.699500 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" podUID="0077aa8d-4c02-40d7-ac71-beb13830dcf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 17:46:59.689690 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:59.689194 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" event={"ID":"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18","Type":"ContainerStarted","Data":"e67287a79cf176a7a4be6fbba0ab119bf0d7ad1b13212e73ec575459a42b6b15"} Nov 02 17:46:59.689690 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:59.689251 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" event={"ID":"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18","Type":"ContainerStarted","Data":"a4fe2af33fe9c3a1793ad00b08c69bd68acd90e799855c81f617a6fe2b5811d8"} Nov 02 17:46:59.698832 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:59.698801 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 17:46:59.698990 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:59.698852 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" podUID="0077aa8d-4c02-40d7-ac71-beb13830dcf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 17:46:59.715481 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:46:59.715421 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" podStartSLOduration=2.715405241 podStartE2EDuration="2.715405241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:59.713298474 +0000 UTC m=+784.404352478" watchObservedRunningTime="2026-11-02 17:46:59.715405241 +0000 UTC m=+784.406459252" Nov 02 17:47:00.357114 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:00.357075 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:47:00.363329 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:00.363298 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:47:00.696520 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:00.696487 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:47:00.697495 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:00.697474 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:47:00.704203 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:00.704177 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:47:00.704489 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:00.704471 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:47:00.705270 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:00.705254 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 17:47:01.007122 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.007033 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69"] Nov 02 17:47:01.011675 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.011610 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.037854 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.037813 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69"] Nov 02 17:47:01.135961 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.135909 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-credential-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.136182 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.135986 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-token\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.136182 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.136158 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkdsw\" (UniqueName: \"kubernetes.io/projected/66c64f99-ea3f-4cef-bac8-372f00ea117c-kube-api-access-lkdsw\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.136304 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.136215 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-workload-certs\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.136304 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.136270 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c64f99-ea3f-4cef-bac8-372f00ea117c-istiod-ca-cert\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.136410 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.136299 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-data\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.136410 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.136381 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-workload-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.136509 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.136441 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-envoy\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.136565 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.136524 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-podinfo\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.238646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.237799 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-workload-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.238646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.237890 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-envoy\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.238646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.237966 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-podinfo\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.238646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.238038 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-credential-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.238646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.238088 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-token\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.238646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.238152 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkdsw\" (UniqueName: \"kubernetes.io/projected/66c64f99-ea3f-4cef-bac8-372f00ea117c-kube-api-access-lkdsw\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.238646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.238204 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-workload-certs\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.238646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.238264 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c64f99-ea3f-4cef-bac8-372f00ea117c-istiod-ca-cert\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.238646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.238295 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-data\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.239268 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.238883 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-data\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.239268 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.239134 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-workload-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.248636 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.247019 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-workload-certs\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.248636 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.247582 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-credential-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.248636 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.248293 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c64f99-ea3f-4cef-bac8-372f00ea117c-istiod-ca-cert\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.289261 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.281239 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-token\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.289261 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.282380 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-envoy\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.294179 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.294142 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-podinfo\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.322652 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.314830 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkdsw\" (UniqueName: \"kubernetes.io/projected/66c64f99-ea3f-4cef-bac8-372f00ea117c-kube-api-access-lkdsw\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.337161 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.336881 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:01.646148 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.646112 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69"] Nov 02 17:47:01.648218 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:47:01.648190 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c64f99_ea3f_4cef_bac8_372f00ea117c.slice/crio-b718dc9d2392a0e7696d0239c4e83108d6e987fb306ca6ae7ff932d87f690bac WatchSource:0}: Error finding container b718dc9d2392a0e7696d0239c4e83108d6e987fb306ca6ae7ff932d87f690bac: Status 404 returned error can't find the container with id b718dc9d2392a0e7696d0239c4e83108d6e987fb306ca6ae7ff932d87f690bac Nov 02 17:47:01.650749 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.650715 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:47:01.650856 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.650778 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:47:01.650856 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.650814 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:47:01.714943 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:01.714906 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" event={"ID":"66c64f99-ea3f-4cef-bac8-372f00ea117c","Type":"ContainerStarted","Data":"b718dc9d2392a0e7696d0239c4e83108d6e987fb306ca6ae7ff932d87f690bac"} Nov 02 17:47:02.722171 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:02.722127 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" event={"ID":"66c64f99-ea3f-4cef-bac8-372f00ea117c","Type":"ContainerStarted","Data":"cf4e2998e0a1bfb743eba102ca1d205b286d6e369397b8467476d55c181777b0"} Nov 02 17:47:02.765660 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:02.765563 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" podStartSLOduration=2.765546956 podStartE2EDuration="2.765546956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:02.762151117 +0000 UTC m=+787.453205160" watchObservedRunningTime="2026-11-02 17:47:02.765546956 +0000 UTC m=+787.456600967" Nov 02 17:47:03.299978 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:03.299940 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n"] Nov 02 17:47:03.304333 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:03.304312 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n" Nov 02 17:47:03.309883 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:03.309833 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-935086-dockercfg-9ckkf\"" Nov 02 17:47:03.314458 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:03.314430 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n"] Nov 02 17:47:03.349034 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:03.337971 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:03.364582 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:03.364549 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-935086\"" Nov 02 17:47:03.364978 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:03.364942 5601 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 17:47:03.365118 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:03.365009 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" podUID="66c64f99-ea3f-4cef-bac8-372f00ea117c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 17:47:03.378555 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:03.378521 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rdd5\" (UniqueName: \"kubernetes.io/projected/1db6feea-b663-4657-bbb2-308d8a1b9fc6-kube-api-access-6rdd5\") pod \"payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n\" (UID: \"1db6feea-b663-4657-bbb2-308d8a1b9fc6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n" Nov 02 17:47:03.378778 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:03.378724 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db6feea-b663-4657-bbb2-308d8a1b9fc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n\" (UID: \"1db6feea-b663-4657-bbb2-308d8a1b9fc6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n" Nov 02 17:47:03.482931 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:03.480709 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db6feea-b663-4657-bbb2-308d8a1b9fc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n\" (UID: \"1db6feea-b663-4657-bbb2-308d8a1b9fc6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n" Nov 02 17:47:03.482931 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:03.480928 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rdd5\" (UniqueName: \"kubernetes.io/projected/1db6feea-b663-4657-bbb2-308d8a1b9fc6-kube-api-access-6rdd5\") pod \"payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n\" (UID: \"1db6feea-b663-4657-bbb2-308d8a1b9fc6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n" Nov 02 17:47:03.482931 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:03.482643 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db6feea-b663-4657-bbb2-308d8a1b9fc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n\" (UID: \"1db6feea-b663-4657-bbb2-308d8a1b9fc6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n" Nov 02 17:47:03.558695 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:03.557358 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rdd5\" (UniqueName: \"kubernetes.io/projected/1db6feea-b663-4657-bbb2-308d8a1b9fc6-kube-api-access-6rdd5\") pod \"payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n\" (UID: \"1db6feea-b663-4657-bbb2-308d8a1b9fc6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n" Nov 02 17:47:03.629680 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:03.627204 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n" Nov 02 17:47:04.006651 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:03.999065 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n"] Nov 02 17:47:04.342699 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:04.341094 5601 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 17:47:04.342699 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:04.341156 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" podUID="66c64f99-ea3f-4cef-bac8-372f00ea117c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 17:47:04.488136 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:47:04.488078 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:47:04.830781 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:04.830663 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n" event={"ID":"1db6feea-b663-4657-bbb2-308d8a1b9fc6","Type":"ContainerStarted","Data":"a12388023033aa95fbb7f6e11bd06370498166dfb8565c586d4ea25630751269"} Nov 02 17:47:04.830781 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:04.830738 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n" event={"ID":"1db6feea-b663-4657-bbb2-308d8a1b9fc6","Type":"ContainerStarted","Data":"c1ba7cc3104f82e7563aa8bc76cff5714bd7ee47c3d3691386225467fe370637"} Nov 02 17:47:04.831630 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:04.831563 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n" Nov 02 17:47:04.860643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:04.857152 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n" podStartSLOduration=1.8571305919999999 podStartE2EDuration="1.857130592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:04.855146538 +0000 UTC m=+789.546200552" watchObservedRunningTime="2026-11-02 17:47:04.857130592 +0000 UTC m=+789.548184607" Nov 02 17:47:05.353645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:05.338038 5601 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 17:47:05.353645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:05.338106 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" podUID="66c64f99-ea3f-4cef-bac8-372f00ea117c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 17:47:06.344640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:06.340720 5601 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 17:47:06.344640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:06.340783 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" podUID="66c64f99-ea3f-4cef-bac8-372f00ea117c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 17:47:06.939644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:06.936653 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg"] Nov 02 17:47:07.007727 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.007084 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg"] Nov 02 17:47:07.007727 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.007217 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.017146 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.017115 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-da5f07e6\"/\"openshift-service-ca.crt\"" Nov 02 17:47:07.019410 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.019385 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-da5f07e6\"/\"kube-root-ca.crt\"" Nov 02 17:47:07.027855 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.023500 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-da5f07e6\"/\"default-dockercfg-w5nzf\"" Nov 02 17:47:07.027855 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.023585 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-da5f07e6\"/\"e2e-lifecycle-model-da5f07e6-kserve-self-signed-certs\"" Nov 02 17:47:07.201710 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.201497 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-home\") pod \"e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.201710 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.201589 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-tls-certs\") pod \"e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.201710 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.201636 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-model-cache\") pod \"e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.202004 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.201924 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-tmp-dir\") pod \"e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.202064 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.202024 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-dshm\") pod \"e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.202121 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.202058 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgzp7\" (UniqueName: \"kubernetes.io/projected/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-kube-api-access-sgzp7\") pod \"e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.308883 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.308213 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-home\") pod \"e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.309217 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.309189 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-tls-certs\") pod \"e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.309384 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.309364 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-model-cache\") pod \"e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.309763 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.309740 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-tmp-dir\") pod \"e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.309934 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.309917 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-dshm\") pod \"e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.310062 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.310045 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgzp7\" (UniqueName: \"kubernetes.io/projected/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-kube-api-access-sgzp7\") pod \"e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.310594 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.308844 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-home\") pod \"e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.311429 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.311403 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-tmp-dir\") pod \"e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.311887 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.311865 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-model-cache\") pod \"e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.324659 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.324214 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgzp7\" (UniqueName: \"kubernetes.io/projected/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-kube-api-access-sgzp7\") pod \"e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.333660 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.331075 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-tls-certs\") pod \"e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.342642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.339009 5601 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 17:47:07.342642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.339082 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" podUID="66c64f99-ea3f-4cef-bac8-372f00ea117c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 17:47:07.342642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.339204 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-dshm\") pod \"e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.356098 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.354950 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:07.646700 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.642608 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl"] Nov 02 17:47:07.721651 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.715231 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl"] Nov 02 17:47:07.721651 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.715358 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.750650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.745982 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.750650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.746049 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.750650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.746100 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.750650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.746145 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.750650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.746195 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.750650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.746291 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2fwx\" (UniqueName: \"kubernetes.io/projected/72a481ce-cb98-41e1-bb69-db80124ba5c4-kube-api-access-g2fwx\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.750650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.746326 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.750650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.746365 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.750650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.746397 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a481ce-cb98-41e1-bb69-db80124ba5c4-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.894645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.878474 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.894645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.878568 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.894645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.878655 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.894645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.878691 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.894645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.878824 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2fwx\" (UniqueName: \"kubernetes.io/projected/72a481ce-cb98-41e1-bb69-db80124ba5c4-kube-api-access-g2fwx\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.894645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.878867 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.894645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.878909 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.894645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.878945 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a481ce-cb98-41e1-bb69-db80124ba5c4-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.894645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.879114 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.894645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.879694 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.894645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.879966 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.894645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.880221 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.894645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.892269 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.894645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.892990 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a481ce-cb98-41e1-bb69-db80124ba5c4-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.898009 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.897927 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.902955 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.902921 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.904008 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.903890 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:07.905006 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:07.904979 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2fwx\" (UniqueName: \"kubernetes.io/projected/72a481ce-cb98-41e1-bb69-db80124ba5c4-kube-api-access-g2fwx\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:08.095641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:08.095344 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:08.129081 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:08.128767 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg"] Nov 02 17:47:08.338483 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:08.338023 5601 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 17:47:08.338483 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:08.338082 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" podUID="66c64f99-ea3f-4cef-bac8-372f00ea117c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 17:47:08.517206 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:08.517176 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl"] Nov 02 17:47:08.518860 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:47:08.518830 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a481ce_cb98_41e1_bb69_db80124ba5c4.slice/crio-eed6ce29ecb8c06f1484efb0ea5ce3f7cccc47b5cd9a2c26a22b508c71f8c935 WatchSource:0}: Error finding container eed6ce29ecb8c06f1484efb0ea5ce3f7cccc47b5cd9a2c26a22b508c71f8c935: Status 404 returned error can't find the container with id eed6ce29ecb8c06f1484efb0ea5ce3f7cccc47b5cd9a2c26a22b508c71f8c935 Nov 02 17:47:08.522015 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:08.521961 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:47:08.522118 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:08.522066 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:47:08.522194 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:08.522130 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:47:08.675173 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:08.675142 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt" Nov 02 17:47:09.031656 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:09.029969 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" event={"ID":"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb","Type":"ContainerStarted","Data":"076593f510bf4a6e9d432857ee8fafdf3a3895a5c4dbf4680b8b56cacdf04b39"} Nov 02 17:47:09.031656 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:09.030022 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" event={"ID":"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb","Type":"ContainerStarted","Data":"f0a1fe003be2fe630020033d33f9a17fc0f9fca586e51c78951e7a1b4af17a14"} Nov 02 17:47:09.070645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:09.070178 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" podStartSLOduration=3.070154044 podStartE2EDuration="3.070154044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:09.05584888 +0000 UTC m=+793.746902905" watchObservedRunningTime="2026-11-02 17:47:09.070154044 +0000 UTC m=+793.761208056" Nov 02 17:47:09.085639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:09.085013 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" event={"ID":"72a481ce-cb98-41e1-bb69-db80124ba5c4","Type":"ContainerStarted","Data":"1fe98daf3a74a709e4afdd3b0c298b6666de9854f14863d2091411d57d1598e5"} Nov 02 17:47:09.085639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:09.085122 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" event={"ID":"72a481ce-cb98-41e1-bb69-db80124ba5c4","Type":"ContainerStarted","Data":"eed6ce29ecb8c06f1484efb0ea5ce3f7cccc47b5cd9a2c26a22b508c71f8c935"} Nov 02 17:47:09.096681 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:09.096637 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:09.099413 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:09.099391 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 17:47:09.100585 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:09.100560 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" podUID="72a481ce-cb98-41e1-bb69-db80124ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 17:47:09.112633 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:09.112556 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" podStartSLOduration=2.112541194 podStartE2EDuration="2.112541194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:09.111373944 +0000 UTC m=+793.802427947" watchObservedRunningTime="2026-11-02 17:47:09.112541194 +0000 UTC m=+793.803595206" Nov 02 17:47:09.339143 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:09.339044 5601 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 17:47:09.339433 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:09.339403 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" podUID="66c64f99-ea3f-4cef-bac8-372f00ea117c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 17:47:09.357796 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:09.357762 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:10.119734 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:10.119687 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 17:47:10.121753 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:10.121713 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" podUID="72a481ce-cb98-41e1-bb69-db80124ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 17:47:10.358060 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:10.358014 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:10.376716 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:10.375084 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:10.390667 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:10.381731 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 17:47:11.096807 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:11.096768 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 17:47:11.096993 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:11.096833 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" podUID="72a481ce-cb98-41e1-bb69-db80124ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 17:47:12.096938 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:12.096904 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 17:47:12.097384 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:12.096958 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" podUID="72a481ce-cb98-41e1-bb69-db80124ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 17:47:13.096517 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:13.096478 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 17:47:13.096895 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:13.096862 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" podUID="72a481ce-cb98-41e1-bb69-db80124ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 17:47:14.096373 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:14.096324 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 17:47:14.096865 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:14.096393 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" podUID="72a481ce-cb98-41e1-bb69-db80124ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 17:47:15.096955 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:15.096922 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 17:47:15.097438 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:15.096980 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" podUID="72a481ce-cb98-41e1-bb69-db80124ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 17:47:15.850931 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:15.850904 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n" Nov 02 17:47:16.096977 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:16.096929 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 17:47:16.097506 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:16.097010 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" podUID="72a481ce-cb98-41e1-bb69-db80124ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 17:47:16.933302 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:16.933264 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596"] Nov 02 17:47:16.936430 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:16.936408 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:16.941046 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:16.941023 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 17:47:16.941046 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:16.941033 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 17:47:16.941046 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:16.941039 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 17:47:16.942063 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:16.942042 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-phftv\"" Nov 02 17:47:16.952905 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:16.952869 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596"] Nov 02 17:47:17.051379 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.051336 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wqx9\" (UniqueName: \"kubernetes.io/projected/f0e27933-7691-4793-922c-4e83bba3914e-kube-api-access-7wqx9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.051574 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.051387 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0e27933-7691-4793-922c-4e83bba3914e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.051574 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.051553 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f0e27933-7691-4793-922c-4e83bba3914e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.051707 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.051593 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f0e27933-7691-4793-922c-4e83bba3914e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.051758 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.051714 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0e27933-7691-4793-922c-4e83bba3914e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.051758 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.051743 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f0e27933-7691-4793-922c-4e83bba3914e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.051857 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.051841 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f0e27933-7691-4793-922c-4e83bba3914e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.106862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.106826 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:17.107610 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.107588 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:17.108313 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.108264 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 17:47:17.153569 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.153530 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0e27933-7691-4793-922c-4e83bba3914e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.153569 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.153580 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f0e27933-7691-4793-922c-4e83bba3914e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.153859 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.153708 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f0e27933-7691-4793-922c-4e83bba3914e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.153859 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.153796 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wqx9\" (UniqueName: \"kubernetes.io/projected/f0e27933-7691-4793-922c-4e83bba3914e-kube-api-access-7wqx9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.153859 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.153822 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0e27933-7691-4793-922c-4e83bba3914e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.154015 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.154007 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f0e27933-7691-4793-922c-4e83bba3914e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.154067 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.154047 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f0e27933-7691-4793-922c-4e83bba3914e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.155483 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.155456 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f0e27933-7691-4793-922c-4e83bba3914e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.158396 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.158364 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f0e27933-7691-4793-922c-4e83bba3914e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.160672 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.160642 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f0e27933-7691-4793-922c-4e83bba3914e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.164362 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.164272 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0e27933-7691-4793-922c-4e83bba3914e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.164362 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.164313 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f0e27933-7691-4793-922c-4e83bba3914e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.173194 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.173159 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0e27933-7691-4793-922c-4e83bba3914e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.176591 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.176559 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wqx9\" (UniqueName: \"kubernetes.io/projected/f0e27933-7691-4793-922c-4e83bba3914e-kube-api-access-7wqx9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dl596\" (UID: \"f0e27933-7691-4793-922c-4e83bba3914e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.219084 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.219001 5601 generic.go:356] "Generic (PLEG): container finished" podID="3f62a981-07b4-4450-9013-959356a33a36" containerID="3c084d3b72fd58de7e76b9b67193e713b508ba8493d39cc76bc9329cedbf059f" exitCode=0 Nov 02 17:47:17.219084 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.219070 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" event={"ID":"3f62a981-07b4-4450-9013-959356a33a36","Type":"ContainerDied","Data":"3c084d3b72fd58de7e76b9b67193e713b508ba8493d39cc76bc9329cedbf059f"} Nov 02 17:47:17.219773 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.219745 5601 scope.go:122] "RemoveContainer" containerID="3c084d3b72fd58de7e76b9b67193e713b508ba8493d39cc76bc9329cedbf059f" Nov 02 17:47:17.221199 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.221150 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:47:17.221313 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.221219 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:47:17.221313 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.221256 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:47:17.252279 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.252024 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:17.375023 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.374990 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:17.396562 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.396247 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:17.576025 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:17.575989 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596"] Nov 02 17:47:17.578097 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:47:17.578063 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e27933_7691_4793_922c_4e83bba3914e.slice/crio-079905f142dc101875e0c532af069204567f91762463cd8c236faa69d2f7f9c2 WatchSource:0}: Error finding container 079905f142dc101875e0c532af069204567f91762463cd8c236faa69d2f7f9c2: Status 404 returned error can't find the container with id 079905f142dc101875e0c532af069204567f91762463cd8c236faa69d2f7f9c2 Nov 02 17:47:18.228346 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:18.228311 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" event={"ID":"f0e27933-7691-4793-922c-4e83bba3914e","Type":"ContainerStarted","Data":"079905f142dc101875e0c532af069204567f91762463cd8c236faa69d2f7f9c2"} Nov 02 17:47:18.230649 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:18.230607 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" event={"ID":"3f62a981-07b4-4450-9013-959356a33a36","Type":"ContainerStarted","Data":"6db0d814f76aa85f6c9cef030d026ce10219c51a1c88850fed2408818a439cad"} Nov 02 17:47:18.273686 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:18.273655 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:47:19.156640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:19.153960 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:47:19.162640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:19.158140 5601 patch_prober.go:29] interesting pod/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:47:19.162640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:19.158221 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" podUID="3f62a981-07b4-4450-9013-959356a33a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 17:47:20.159366 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:20.159323 5601 patch_prober.go:29] interesting pod/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:47:20.159874 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:20.159392 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" podUID="3f62a981-07b4-4450-9013-959356a33a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 17:47:21.157707 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:21.157117 5601 patch_prober.go:29] interesting pod/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:47:21.157707 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:21.157183 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" podUID="3f62a981-07b4-4450-9013-959356a33a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 17:47:21.767101 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:21.767060 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:47:21.767406 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:21.767129 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:47:22.161543 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:22.157025 5601 patch_prober.go:29] interesting pod/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:47:22.161543 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:22.157093 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" podUID="3f62a981-07b4-4450-9013-959356a33a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 17:47:22.327786 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:22.327747 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" event={"ID":"f0e27933-7691-4793-922c-4e83bba3914e","Type":"ContainerStarted","Data":"de103fb87bed7fc86fecc07f266afa7cb82f6dd2972011b87fb816359fc65728"} Nov 02 17:47:22.328040 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:22.328023 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:22.333014 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:22.332970 5601 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dl596 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:47:22.333146 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:22.333033 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" podUID="f0e27933-7691-4793-922c-4e83bba3914e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:47:23.160247 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:23.160209 5601 patch_prober.go:29] interesting pod/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:47:23.160902 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:23.160860 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" podUID="3f62a981-07b4-4450-9013-959356a33a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 17:47:23.352198 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:23.351883 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" Nov 02 17:47:23.380358 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:23.380304 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dl596" podStartSLOduration=3.19481413 podStartE2EDuration="7.380288843s" totalImagesPullingTime="4.185474713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:22.384874137 +0000 UTC m=+807.075928149" watchObservedRunningTime="2026-11-02 17:47:23.380288843 +0000 UTC m=+808.071342855" Nov 02 17:47:23.443880 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:23.443845 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:47:24.157791 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:24.157756 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:47:24.345581 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:24.345549 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:47:34.485323 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:47:34.485287 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:47:38.852314 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:47:38.852271 5601 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe64c3d_8ff3_400f_8dc9_d7c8edcdbb18.slice/crio-e67287a79cf176a7a4be6fbba0ab119bf0d7ad1b13212e73ec575459a42b6b15.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@4-19979-0.service\": RecentStats: unable to find data in memory cache]" Nov 02 17:47:39.489985 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:39.489933 5601 generic.go:356] "Generic (PLEG): container finished" podID="dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" containerID="e67287a79cf176a7a4be6fbba0ab119bf0d7ad1b13212e73ec575459a42b6b15" exitCode=0 Nov 02 17:47:39.490223 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:39.490197 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" event={"ID":"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18","Type":"ContainerDied","Data":"e67287a79cf176a7a4be6fbba0ab119bf0d7ad1b13212e73ec575459a42b6b15"} Nov 02 17:47:39.490801 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:39.490780 5601 scope.go:122] "RemoveContainer" containerID="e67287a79cf176a7a4be6fbba0ab119bf0d7ad1b13212e73ec575459a42b6b15" Nov 02 17:47:39.493189 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:39.493156 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:47:39.493290 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:39.493259 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:47:39.493361 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:39.493323 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:47:40.502898 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:40.502862 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" event={"ID":"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18","Type":"ContainerStarted","Data":"710daf26fe0fff570088b2f96ddb16e42b83b48b871c621a2f5f372665974b5a"} Nov 02 17:47:41.363242 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:41.358906 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:47:41.364969 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:41.364928 5601 patch_prober.go:29] interesting pod/e2e-api-b-935086-openshift-default-8497785c69-j2mzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 17:47:41.365112 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:41.365006 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" podUID="dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 17:47:42.358640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:42.358014 5601 patch_prober.go:29] interesting pod/e2e-api-b-935086-openshift-default-8497785c69-j2mzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 17:47:42.358640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:42.358089 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" podUID="dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 17:47:43.358794 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:43.358750 5601 patch_prober.go:29] interesting pod/e2e-api-b-935086-openshift-default-8497785c69-j2mzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 17:47:43.359331 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:43.358825 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" podUID="dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 17:47:44.359075 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:44.359041 5601 patch_prober.go:29] interesting pod/e2e-api-b-935086-openshift-default-8497785c69-j2mzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 17:47:44.359518 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:44.359121 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" podUID="dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 17:47:44.483666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:44.482865 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh"] Nov 02 17:47:44.483666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:44.483104 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" podUID="aa7f1f3d-5879-4234-82f0-02f0100f199f" containerName="maas-api" containerID="cri-o://6dd7ab510bc4f9ee510eeb90fcd19130d5596b9ad0db1a96921781db800c7c54" gracePeriod=30 Nov 02 17:47:44.537661 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:44.537053 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" podUID="aa7f1f3d-5879-4234-82f0-02f0100f199f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.82:8443/health\": dial tcp 10.134.0.82:8443: connect: connection refused" Nov 02 17:47:45.164699 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.160817 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" Nov 02 17:47:45.219292 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.219250 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa7f1f3d-5879-4234-82f0-02f0100f199f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa7f1f3d-5879-4234-82f0-02f0100f199f-metrics-certs\") pod \"aa7f1f3d-5879-4234-82f0-02f0100f199f\" (UID: \"aa7f1f3d-5879-4234-82f0-02f0100f199f\") " Nov 02 17:47:45.219689 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.219666 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa7f1f3d-5879-4234-82f0-02f0100f199f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa7f1f3d-5879-4234-82f0-02f0100f199f-maas-api-tls\") pod \"aa7f1f3d-5879-4234-82f0-02f0100f199f\" (UID: \"aa7f1f3d-5879-4234-82f0-02f0100f199f\") " Nov 02 17:47:45.219927 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.219909 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa7f1f3d-5879-4234-82f0-02f0100f199f-kube-api-access-kbt2h\" (UniqueName: \"kubernetes.io/projected/aa7f1f3d-5879-4234-82f0-02f0100f199f-kube-api-access-kbt2h\") pod \"aa7f1f3d-5879-4234-82f0-02f0100f199f\" (UID: \"aa7f1f3d-5879-4234-82f0-02f0100f199f\") " Nov 02 17:47:45.220069 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.220054 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa7f1f3d-5879-4234-82f0-02f0100f199f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7f1f3d-5879-4234-82f0-02f0100f199f-openshift-service-ca\") pod \"aa7f1f3d-5879-4234-82f0-02f0100f199f\" (UID: \"aa7f1f3d-5879-4234-82f0-02f0100f199f\") " Nov 02 17:47:45.222409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.222382 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7f1f3d-5879-4234-82f0-02f0100f199f-openshift-service-ca" pod "aa7f1f3d-5879-4234-82f0-02f0100f199f" (UID: "aa7f1f3d-5879-4234-82f0-02f0100f199f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:47:45.278660 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.277899 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa7f1f3d-5879-4234-82f0-02f0100f199f-metrics-certs" pod "aa7f1f3d-5879-4234-82f0-02f0100f199f" (UID: "aa7f1f3d-5879-4234-82f0-02f0100f199f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:47:45.278660 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.277921 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa7f1f3d-5879-4234-82f0-02f0100f199f-maas-api-tls" pod "aa7f1f3d-5879-4234-82f0-02f0100f199f" (UID: "aa7f1f3d-5879-4234-82f0-02f0100f199f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:47:45.283734 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.283685 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7f1f3d-5879-4234-82f0-02f0100f199f-kube-api-access-kbt2h" pod "aa7f1f3d-5879-4234-82f0-02f0100f199f" (UID: "aa7f1f3d-5879-4234-82f0-02f0100f199f"). InnerVolumeSpecName "kube-api-access-kbt2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:45.322264 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.322222 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa7f1f3d-5879-4234-82f0-02f0100f199f-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:47:45.322264 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.322264 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa7f1f3d-5879-4234-82f0-02f0100f199f-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:47:45.322465 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.322305 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbt2h\" (UniqueName: \"kubernetes.io/projected/aa7f1f3d-5879-4234-82f0-02f0100f199f-kube-api-access-kbt2h\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:47:45.322465 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.322321 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7f1f3d-5879-4234-82f0-02f0100f199f-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:47:45.361352 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.361299 5601 patch_prober.go:29] interesting pod/e2e-api-b-935086-openshift-default-8497785c69-j2mzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 17:47:45.361928 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.361761 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" podUID="dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 17:47:45.578049 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.576379 5601 generic.go:356] "Generic (PLEG): container finished" podID="aa7f1f3d-5879-4234-82f0-02f0100f199f" containerID="6dd7ab510bc4f9ee510eeb90fcd19130d5596b9ad0db1a96921781db800c7c54" exitCode=0 Nov 02 17:47:45.578049 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.576548 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" event={"ID":"aa7f1f3d-5879-4234-82f0-02f0100f199f","Type":"ContainerDied","Data":"6dd7ab510bc4f9ee510eeb90fcd19130d5596b9ad0db1a96921781db800c7c54"} Nov 02 17:47:45.578049 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.576603 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" event={"ID":"aa7f1f3d-5879-4234-82f0-02f0100f199f","Type":"ContainerDied","Data":"166ecce34b806cd7d93e51c6e6bb90766d355a9a5838c728718036d469d2f8db"} Nov 02 17:47:45.578049 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.576657 5601 scope.go:122] "RemoveContainer" containerID="6dd7ab510bc4f9ee510eeb90fcd19130d5596b9ad0db1a96921781db800c7c54" Nov 02 17:47:45.578049 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.576881 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh" Nov 02 17:47:45.642685 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.641689 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh"] Nov 02 17:47:45.656782 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.655905 5601 scope.go:122] "RemoveContainer" containerID="6dd7ab510bc4f9ee510eeb90fcd19130d5596b9ad0db1a96921781db800c7c54" Nov 02 17:47:45.664637 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:47:45.660975 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dd7ab510bc4f9ee510eeb90fcd19130d5596b9ad0db1a96921781db800c7c54\": container with ID starting with 6dd7ab510bc4f9ee510eeb90fcd19130d5596b9ad0db1a96921781db800c7c54 not found: ID does not exist" containerID="6dd7ab510bc4f9ee510eeb90fcd19130d5596b9ad0db1a96921781db800c7c54" Nov 02 17:47:45.664637 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.661023 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd7ab510bc4f9ee510eeb90fcd19130d5596b9ad0db1a96921781db800c7c54"} err="failed to get container status \"6dd7ab510bc4f9ee510eeb90fcd19130d5596b9ad0db1a96921781db800c7c54\": rpc error: code = NotFound desc = could not find container \"6dd7ab510bc4f9ee510eeb90fcd19130d5596b9ad0db1a96921781db800c7c54\": container with ID starting with 6dd7ab510bc4f9ee510eeb90fcd19130d5596b9ad0db1a96921781db800c7c54 not found: ID does not exist" Nov 02 17:47:45.685638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.667298 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2893d1-86dbfd945d-q2dmh"] Nov 02 17:47:45.709641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:45.697732 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:47:46.364417 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:46.364379 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:47:46.598055 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:46.597779 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:47:47.367596 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:47.365384 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt"] Nov 02 17:47:48.231636 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:48.216734 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa7f1f3d-5879-4234-82f0-02f0100f199f" path="/var/lib/kubelet/pods/aa7f1f3d-5879-4234-82f0-02f0100f199f/volumes" Nov 02 17:47:48.317304 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:48.317259 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" podUID="3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:52.373508 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:52.373458 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt" podUID="c470dc21-8297-4937-9589-5cf3a292741b" containerName="payload-pre-processing" containerID="cri-o://c13ce7b5a9efd8ab1ccc1bc4dc72827db6825db88629ac45f103d94691508e30" gracePeriod=25 Nov 02 17:47:52.667823 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:52.667792 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt" Nov 02 17:47:52.693124 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:52.693091 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c470dc21-8297-4937-9589-5cf3a292741b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c470dc21-8297-4937-9589-5cf3a292741b-plugins-config-volume\") pod \"c470dc21-8297-4937-9589-5cf3a292741b\" (UID: \"c470dc21-8297-4937-9589-5cf3a292741b\") " Nov 02 17:47:52.693310 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:52.693181 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c470dc21-8297-4937-9589-5cf3a292741b-kube-api-access-l6cn8\" (UniqueName: \"kubernetes.io/projected/c470dc21-8297-4937-9589-5cf3a292741b-kube-api-access-l6cn8\") pod \"c470dc21-8297-4937-9589-5cf3a292741b\" (UID: \"c470dc21-8297-4937-9589-5cf3a292741b\") " Nov 02 17:47:52.693991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:52.693948 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c470dc21-8297-4937-9589-5cf3a292741b-plugins-config-volume" pod "c470dc21-8297-4937-9589-5cf3a292741b" (UID: "c470dc21-8297-4937-9589-5cf3a292741b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:47:52.696355 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:52.696329 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c470dc21-8297-4937-9589-5cf3a292741b-kube-api-access-l6cn8" pod "c470dc21-8297-4937-9589-5cf3a292741b" (UID: "c470dc21-8297-4937-9589-5cf3a292741b"). InnerVolumeSpecName "kube-api-access-l6cn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:52.795272 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:52.795240 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c470dc21-8297-4937-9589-5cf3a292741b-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:47:52.795272 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:52.795268 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6cn8\" (UniqueName: \"kubernetes.io/projected/c470dc21-8297-4937-9589-5cf3a292741b-kube-api-access-l6cn8\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:47:53.217315 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:53.217279 5601 generic.go:356] "Generic (PLEG): container finished" podID="c470dc21-8297-4937-9589-5cf3a292741b" containerID="c13ce7b5a9efd8ab1ccc1bc4dc72827db6825db88629ac45f103d94691508e30" exitCode=0 Nov 02 17:47:53.217489 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:53.217321 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt" Nov 02 17:47:53.217489 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:53.217356 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt" event={"ID":"c470dc21-8297-4937-9589-5cf3a292741b","Type":"ContainerDied","Data":"c13ce7b5a9efd8ab1ccc1bc4dc72827db6825db88629ac45f103d94691508e30"} Nov 02 17:47:53.217489 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:53.217427 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt" event={"ID":"c470dc21-8297-4937-9589-5cf3a292741b","Type":"ContainerDied","Data":"6478fe2adb2fcc52a63c04dec514c8ed439737f992eb8845990c81a53d12db89"} Nov 02 17:47:53.217489 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:53.217452 5601 scope.go:122] "RemoveContainer" containerID="c13ce7b5a9efd8ab1ccc1bc4dc72827db6825db88629ac45f103d94691508e30" Nov 02 17:47:53.228171 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:53.228135 5601 scope.go:122] "RemoveContainer" containerID="c13ce7b5a9efd8ab1ccc1bc4dc72827db6825db88629ac45f103d94691508e30" Nov 02 17:47:53.228471 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:47:53.228453 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c13ce7b5a9efd8ab1ccc1bc4dc72827db6825db88629ac45f103d94691508e30\": container with ID starting with c13ce7b5a9efd8ab1ccc1bc4dc72827db6825db88629ac45f103d94691508e30 not found: ID does not exist" containerID="c13ce7b5a9efd8ab1ccc1bc4dc72827db6825db88629ac45f103d94691508e30" Nov 02 17:47:53.228528 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:53.228477 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c13ce7b5a9efd8ab1ccc1bc4dc72827db6825db88629ac45f103d94691508e30"} err="failed to get container status \"c13ce7b5a9efd8ab1ccc1bc4dc72827db6825db88629ac45f103d94691508e30\": rpc error: code = NotFound desc = could not find container \"c13ce7b5a9efd8ab1ccc1bc4dc72827db6825db88629ac45f103d94691508e30\": container with ID starting with c13ce7b5a9efd8ab1ccc1bc4dc72827db6825db88629ac45f103d94691508e30 not found: ID does not exist" Nov 02 17:47:53.241524 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:53.241502 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt"] Nov 02 17:47:53.249093 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:53.249065 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-58970f-76bf8f9875-lfqpt"] Nov 02 17:47:53.426287 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:53.426253 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c470dc21-8297-4937-9589-5cf3a292741b" path="/var/lib/kubelet/pods/c470dc21-8297-4937-9589-5cf3a292741b/volumes" Nov 02 17:47:58.750568 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:47:58.750532 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n"] Nov 02 17:48:03.751551 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:03.751490 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n" podUID="1db6feea-b663-4657-bbb2-308d8a1b9fc6" containerName="payload-pre-processing" containerID="cri-o://a12388023033aa95fbb7f6e11bd06370498166dfb8565c586d4ea25630751269" gracePeriod=25 Nov 02 17:48:04.016674 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:04.016647 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n" Nov 02 17:48:04.110161 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:04.110129 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db6feea-b663-4657-bbb2-308d8a1b9fc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db6feea-b663-4657-bbb2-308d8a1b9fc6-plugins-config-volume\") pod \"1db6feea-b663-4657-bbb2-308d8a1b9fc6\" (UID: \"1db6feea-b663-4657-bbb2-308d8a1b9fc6\") " Nov 02 17:48:04.110335 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:04.110233 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db6feea-b663-4657-bbb2-308d8a1b9fc6-kube-api-access-6rdd5\" (UniqueName: \"kubernetes.io/projected/1db6feea-b663-4657-bbb2-308d8a1b9fc6-kube-api-access-6rdd5\") pod \"1db6feea-b663-4657-bbb2-308d8a1b9fc6\" (UID: \"1db6feea-b663-4657-bbb2-308d8a1b9fc6\") " Nov 02 17:48:04.110489 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:04.110466 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db6feea-b663-4657-bbb2-308d8a1b9fc6-plugins-config-volume" pod "1db6feea-b663-4657-bbb2-308d8a1b9fc6" (UID: "1db6feea-b663-4657-bbb2-308d8a1b9fc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:48:04.110818 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:04.110801 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db6feea-b663-4657-bbb2-308d8a1b9fc6-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:04.112439 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:04.112418 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db6feea-b663-4657-bbb2-308d8a1b9fc6-kube-api-access-6rdd5" pod "1db6feea-b663-4657-bbb2-308d8a1b9fc6" (UID: "1db6feea-b663-4657-bbb2-308d8a1b9fc6"). InnerVolumeSpecName "kube-api-access-6rdd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:04.212243 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:04.212204 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rdd5\" (UniqueName: \"kubernetes.io/projected/1db6feea-b663-4657-bbb2-308d8a1b9fc6-kube-api-access-6rdd5\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:04.434873 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:04.434834 5601 generic.go:356] "Generic (PLEG): container finished" podID="1db6feea-b663-4657-bbb2-308d8a1b9fc6" containerID="a12388023033aa95fbb7f6e11bd06370498166dfb8565c586d4ea25630751269" exitCode=0 Nov 02 17:48:04.435094 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:04.434914 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n" Nov 02 17:48:04.435094 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:04.434933 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n" event={"ID":"1db6feea-b663-4657-bbb2-308d8a1b9fc6","Type":"ContainerDied","Data":"a12388023033aa95fbb7f6e11bd06370498166dfb8565c586d4ea25630751269"} Nov 02 17:48:04.435094 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:04.434998 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n" event={"ID":"1db6feea-b663-4657-bbb2-308d8a1b9fc6","Type":"ContainerDied","Data":"c1ba7cc3104f82e7563aa8bc76cff5714bd7ee47c3d3691386225467fe370637"} Nov 02 17:48:04.435094 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:04.435013 5601 scope.go:122] "RemoveContainer" containerID="a12388023033aa95fbb7f6e11bd06370498166dfb8565c586d4ea25630751269" Nov 02 17:48:04.446389 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:04.446369 5601 scope.go:122] "RemoveContainer" containerID="a12388023033aa95fbb7f6e11bd06370498166dfb8565c586d4ea25630751269" Nov 02 17:48:04.446697 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:48:04.446672 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a12388023033aa95fbb7f6e11bd06370498166dfb8565c586d4ea25630751269\": container with ID starting with a12388023033aa95fbb7f6e11bd06370498166dfb8565c586d4ea25630751269 not found: ID does not exist" containerID="a12388023033aa95fbb7f6e11bd06370498166dfb8565c586d4ea25630751269" Nov 02 17:48:04.446782 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:04.446703 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a12388023033aa95fbb7f6e11bd06370498166dfb8565c586d4ea25630751269"} err="failed to get container status \"a12388023033aa95fbb7f6e11bd06370498166dfb8565c586d4ea25630751269\": rpc error: code = NotFound desc = could not find container \"a12388023033aa95fbb7f6e11bd06370498166dfb8565c586d4ea25630751269\": container with ID starting with a12388023033aa95fbb7f6e11bd06370498166dfb8565c586d4ea25630751269 not found: ID does not exist" Nov 02 17:48:04.465538 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:04.464384 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n"] Nov 02 17:48:04.468405 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:04.468380 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-935086-557dbc4555-g9w9n"] Nov 02 17:48:04.491574 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:48:04.491534 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:48:05.427378 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:05.427333 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db6feea-b663-4657-bbb2-308d8a1b9fc6" path="/var/lib/kubelet/pods/1db6feea-b663-4657-bbb2-308d8a1b9fc6/volumes" Nov 02 17:48:07.252024 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:07.251988 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw"] Nov 02 17:48:07.252595 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:07.252297 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" podUID="dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" containerName="istio-proxy" containerID="cri-o://710daf26fe0fff570088b2f96ddb16e42b83b48b871c621a2f5f372665974b5a" gracePeriod=30 Nov 02 17:48:12.567139 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.565113 5601 generic.go:356] "Generic (PLEG): container finished" podID="dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" containerID="710daf26fe0fff570088b2f96ddb16e42b83b48b871c621a2f5f372665974b5a" exitCode=0 Nov 02 17:48:12.567139 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.565178 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" event={"ID":"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18","Type":"ContainerDied","Data":"710daf26fe0fff570088b2f96ddb16e42b83b48b871c621a2f5f372665974b5a"} Nov 02 17:48:12.567139 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.565235 5601 scope.go:122] "RemoveContainer" containerID="e67287a79cf176a7a4be6fbba0ab119bf0d7ad1b13212e73ec575459a42b6b15" Nov 02 17:48:12.599553 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.599524 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:48:12.751671 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.751638 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-envoy\") pod \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " Nov 02 17:48:12.751868 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.751761 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-data\") pod \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " Nov 02 17:48:12.751868 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.751813 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-podinfo\") pod \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " Nov 02 17:48:12.751991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.751884 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-workload-certs\") pod \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " Nov 02 17:48:12.751991 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.751949 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-credential-socket\") pod \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " Nov 02 17:48:12.752093 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.751994 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-workload-socket\") pod \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " Nov 02 17:48:12.752093 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.752037 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istiod-ca-cert\") pod \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " Nov 02 17:48:12.752093 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.752050 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-data" pod "dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" (UID: "dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:12.752093 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.752076 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-kube-api-access-j7xjm\" (UniqueName: \"kubernetes.io/projected/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-kube-api-access-j7xjm\") pod \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " Nov 02 17:48:12.752380 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.752236 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-token\" (UniqueName: \"kubernetes.io/projected/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-token\") pod \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\" (UID: \"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18\") " Nov 02 17:48:12.752380 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.752232 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-workload-certs" pod "dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" (UID: "dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:12.752380 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.752291 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-workload-socket" pod "dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" (UID: "dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:12.752654 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.752629 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istiod-ca-cert" pod "dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" (UID: "dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:48:12.753015 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.752989 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:12.753015 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.753016 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:12.753273 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.753031 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:12.753273 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.753045 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:12.753273 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.753236 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-credential-socket" pod "dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" (UID: "dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:12.754723 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.754395 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-envoy" pod "dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" (UID: "dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:12.781637 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.780880 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-kube-api-access-j7xjm" pod "dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" (UID: "dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18"). InnerVolumeSpecName "kube-api-access-j7xjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:12.815644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.796829 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-token" pod "dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" (UID: "dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:12.821087 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.819754 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-podinfo" pod "dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" (UID: "dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:48:12.870641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.862067 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:12.870641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.862107 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:12.870641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.862124 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7xjm\" (UniqueName: \"kubernetes.io/projected/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-kube-api-access-j7xjm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:12.870641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.862143 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:12.870641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:12.862155 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:13.615687 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:13.603976 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" event={"ID":"dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18","Type":"ContainerDied","Data":"a4fe2af33fe9c3a1793ad00b08c69bd68acd90e799855c81f617a6fe2b5811d8"} Nov 02 17:48:13.615687 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:13.604051 5601 scope.go:122] "RemoveContainer" containerID="710daf26fe0fff570088b2f96ddb16e42b83b48b871c621a2f5f372665974b5a" Nov 02 17:48:13.615687 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:13.604210 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw" Nov 02 17:48:13.761638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:13.754184 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw"] Nov 02 17:48:13.761638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:13.759726 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-935086-openshift-default-8497785c69-j2mzw"] Nov 02 17:48:15.431159 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:15.431125 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" path="/var/lib/kubelet/pods/dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18/volumes" Nov 02 17:48:34.485142 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:48:34.485094 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:48:43.002717 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.002679 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64"] Nov 02 17:48:43.004180 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.004154 5601 cpu_manager.go:414] "containerMap: removing container" podUID="aa7f1f3d-5879-4234-82f0-02f0100f199f" containerName="maas-api" Nov 02 17:48:43.004350 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.004335 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa7f1f3d-5879-4234-82f0-02f0100f199f" containerName="maas-api" Nov 02 17:48:43.004461 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.004447 5601 cpu_manager.go:414] "containerMap: removing container" podUID="c470dc21-8297-4937-9589-5cf3a292741b" containerName="payload-pre-processing" Nov 02 17:48:43.004546 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.004534 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c470dc21-8297-4937-9589-5cf3a292741b" containerName="payload-pre-processing" Nov 02 17:48:43.004687 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.004674 5601 cpu_manager.go:414] "containerMap: removing container" podUID="dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" containerName="istio-proxy" Nov 02 17:48:43.004785 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.004771 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" containerName="istio-proxy" Nov 02 17:48:43.004876 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.004864 5601 cpu_manager.go:414] "containerMap: removing container" podUID="1db6feea-b663-4657-bbb2-308d8a1b9fc6" containerName="payload-pre-processing" Nov 02 17:48:43.004958 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.004946 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db6feea-b663-4657-bbb2-308d8a1b9fc6" containerName="payload-pre-processing" Nov 02 17:48:43.005054 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.005044 5601 cpu_manager.go:414] "containerMap: removing container" podUID="dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" containerName="istio-proxy" Nov 02 17:48:43.005127 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.005117 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" containerName="istio-proxy" Nov 02 17:48:43.005417 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.005402 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" containerName="istio-proxy" Nov 02 17:48:43.005523 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.005512 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db6feea-b663-4657-bbb2-308d8a1b9fc6" containerName="payload-pre-processing" Nov 02 17:48:43.005599 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.005590 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe64c3d-8ff3-400f-8dc9-d7c8edcdbb18" containerName="istio-proxy" Nov 02 17:48:43.005714 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.005702 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa7f1f3d-5879-4234-82f0-02f0100f199f" containerName="maas-api" Nov 02 17:48:43.005807 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.005793 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="c470dc21-8297-4937-9589-5cf3a292741b" containerName="payload-pre-processing" Nov 02 17:48:43.013390 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.009900 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.025359 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.025129 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a976bc-openshift-default-dockercfg-qwjsm\"" Nov 02 17:48:43.072690 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.072177 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-data\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.072690 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.072308 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-podinfo\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.072690 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.072369 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-token\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.072690 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.072415 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-workload-certs\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.072690 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.072449 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-credential-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.072690 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.072481 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbmkq\" (UniqueName: \"kubernetes.io/projected/61248c0c-a11a-4d6b-b273-2abb5d1434d6-kube-api-access-mbmkq\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.072690 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.072512 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-workload-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.072690 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.072551 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-envoy\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.072690 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.072577 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istiod-ca-cert\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.078905 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.078878 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64"] Nov 02 17:48:43.174676 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.174643 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-data\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.174885 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.174746 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-podinfo\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.174885 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.174787 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-token\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.175019 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.174921 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-workload-certs\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.175019 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.174977 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-credential-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.175133 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.175019 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbmkq\" (UniqueName: \"kubernetes.io/projected/61248c0c-a11a-4d6b-b273-2abb5d1434d6-kube-api-access-mbmkq\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.175133 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.175053 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-workload-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.175133 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.175090 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-envoy\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.175133 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.175113 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istiod-ca-cert\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.175344 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.175139 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-data\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.175577 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.175548 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-workload-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.175750 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.175592 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-workload-certs\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.175750 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.175739 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-credential-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.176250 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.176228 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istiod-ca-cert\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.177879 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.177847 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-envoy\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.178069 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.178052 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-podinfo\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.184337 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.184312 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbmkq\" (UniqueName: \"kubernetes.io/projected/61248c0c-a11a-4d6b-b273-2abb5d1434d6-kube-api-access-mbmkq\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.184832 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.184812 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-token\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-njg64\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.379248 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.378915 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:43.688856 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.688829 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64"] Nov 02 17:48:43.690039 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:48:43.690008 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61248c0c_a11a_4d6b_b273_2abb5d1434d6.slice/crio-f547402187a21420c3888bab9bb4b637f18ffaeaa0a3423e327668aa0433848e WatchSource:0}: Error finding container f547402187a21420c3888bab9bb4b637f18ffaeaa0a3423e327668aa0433848e: Status 404 returned error can't find the container with id f547402187a21420c3888bab9bb4b637f18ffaeaa0a3423e327668aa0433848e Nov 02 17:48:43.695852 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.695795 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:48:43.696098 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.696075 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:48:43.696265 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.696241 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:48:43.906594 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.906554 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" event={"ID":"61248c0c-a11a-4d6b-b273-2abb5d1434d6","Type":"ContainerStarted","Data":"0fd810821e350a86c599013ab0c9ad0658138d24274886a0b4ce01e2617f422a"} Nov 02 17:48:43.906796 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.906665 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" event={"ID":"61248c0c-a11a-4d6b-b273-2abb5d1434d6","Type":"ContainerStarted","Data":"f547402187a21420c3888bab9bb4b637f18ffaeaa0a3423e327668aa0433848e"} Nov 02 17:48:43.932398 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:43.932344 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" podStartSLOduration=1.9323276169999999 podStartE2EDuration="1.932327617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:43.928329075 +0000 UTC m=+888.619383088" watchObservedRunningTime="2026-11-02 17:48:43.932327617 +0000 UTC m=+888.623381628" Nov 02 17:48:44.380479 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:44.380440 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:44.382047 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:44.382023 5601 patch_prober.go:29] interesting pod/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 17:48:44.382194 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:44.382066 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" podUID="61248c0c-a11a-4d6b-b273-2abb5d1434d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 17:48:45.380493 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:45.380456 5601 patch_prober.go:29] interesting pod/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 17:48:45.380946 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:45.380512 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" podUID="61248c0c-a11a-4d6b-b273-2abb5d1434d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 17:48:46.380477 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:46.380438 5601 patch_prober.go:29] interesting pod/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 17:48:46.380697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:46.380505 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" podUID="61248c0c-a11a-4d6b-b273-2abb5d1434d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 17:48:46.604463 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:46.602912 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64"] Nov 02 17:48:46.946973 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:46.942270 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" podUID="61248c0c-a11a-4d6b-b273-2abb5d1434d6" containerName="istio-proxy" containerID="cri-o://0fd810821e350a86c599013ab0c9ad0658138d24274886a0b4ce01e2617f422a" gracePeriod=30 Nov 02 17:48:55.019687 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.019644 5601 generic.go:356] "Generic (PLEG): container finished" podID="61248c0c-a11a-4d6b-b273-2abb5d1434d6" containerID="0fd810821e350a86c599013ab0c9ad0658138d24274886a0b4ce01e2617f422a" exitCode=0 Nov 02 17:48:55.020178 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.019707 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" event={"ID":"61248c0c-a11a-4d6b-b273-2abb5d1434d6","Type":"ContainerDied","Data":"0fd810821e350a86c599013ab0c9ad0658138d24274886a0b4ce01e2617f422a"} Nov 02 17:48:55.204220 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.204191 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:55.255367 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.255282 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istiod-ca-cert\") pod \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " Nov 02 17:48:55.255367 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.255339 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-token\" (UniqueName: \"kubernetes.io/projected/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-token\") pod \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " Nov 02 17:48:55.255606 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.255373 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-data\") pod \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " Nov 02 17:48:55.255606 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.255453 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61248c0c-a11a-4d6b-b273-2abb5d1434d6-kube-api-access-mbmkq\" (UniqueName: \"kubernetes.io/projected/61248c0c-a11a-4d6b-b273-2abb5d1434d6-kube-api-access-mbmkq\") pod \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " Nov 02 17:48:55.255606 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.255501 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-envoy\") pod \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " Nov 02 17:48:55.255606 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.255523 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-podinfo\") pod \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " Nov 02 17:48:55.255606 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.255583 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-workload-certs\") pod \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " Nov 02 17:48:55.255924 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.255703 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-data" pod "61248c0c-a11a-4d6b-b273-2abb5d1434d6" (UID: "61248c0c-a11a-4d6b-b273-2abb5d1434d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:55.255924 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.255701 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istiod-ca-cert" pod "61248c0c-a11a-4d6b-b273-2abb5d1434d6" (UID: "61248c0c-a11a-4d6b-b273-2abb5d1434d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:48:55.255924 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.255754 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-workload-socket\") pod \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " Nov 02 17:48:55.255924 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.255799 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-credential-socket\") pod \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\" (UID: \"61248c0c-a11a-4d6b-b273-2abb5d1434d6\") " Nov 02 17:48:55.256170 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.256071 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-workload-certs" pod "61248c0c-a11a-4d6b-b273-2abb5d1434d6" (UID: "61248c0c-a11a-4d6b-b273-2abb5d1434d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:55.256285 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.256263 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-credential-socket" pod "61248c0c-a11a-4d6b-b273-2abb5d1434d6" (UID: "61248c0c-a11a-4d6b-b273-2abb5d1434d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:55.256442 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.256425 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-workload-socket" pod "61248c0c-a11a-4d6b-b273-2abb5d1434d6" (UID: "61248c0c-a11a-4d6b-b273-2abb5d1434d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:55.256548 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.256532 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:55.256609 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.256551 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:55.256609 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.256561 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:55.256609 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.256570 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:55.256609 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.256579 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:55.257885 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.257862 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-token" pod "61248c0c-a11a-4d6b-b273-2abb5d1434d6" (UID: "61248c0c-a11a-4d6b-b273-2abb5d1434d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:55.258130 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.258108 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-envoy" pod "61248c0c-a11a-4d6b-b273-2abb5d1434d6" (UID: "61248c0c-a11a-4d6b-b273-2abb5d1434d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:55.258238 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.258133 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-podinfo" pod "61248c0c-a11a-4d6b-b273-2abb5d1434d6" (UID: "61248c0c-a11a-4d6b-b273-2abb5d1434d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:48:55.258516 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.258497 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61248c0c-a11a-4d6b-b273-2abb5d1434d6-kube-api-access-mbmkq" pod "61248c0c-a11a-4d6b-b273-2abb5d1434d6" (UID: "61248c0c-a11a-4d6b-b273-2abb5d1434d6"). InnerVolumeSpecName "kube-api-access-mbmkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:55.358420 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.358383 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbmkq\" (UniqueName: \"kubernetes.io/projected/61248c0c-a11a-4d6b-b273-2abb5d1434d6-kube-api-access-mbmkq\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:55.358420 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.358418 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:55.358420 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.358427 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:55.358685 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:55.358438 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61248c0c-a11a-4d6b-b273-2abb5d1434d6-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:56.028367 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:56.028336 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" Nov 02 17:48:56.028367 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:56.028360 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64" event={"ID":"61248c0c-a11a-4d6b-b273-2abb5d1434d6","Type":"ContainerDied","Data":"f547402187a21420c3888bab9bb4b637f18ffaeaa0a3423e327668aa0433848e"} Nov 02 17:48:56.028954 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:56.028426 5601 scope.go:122] "RemoveContainer" containerID="0fd810821e350a86c599013ab0c9ad0658138d24274886a0b4ce01e2617f422a" Nov 02 17:48:56.056605 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:56.056576 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64"] Nov 02 17:48:56.063130 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:56.063094 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-njg64"] Nov 02 17:48:57.426712 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:48:57.426678 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61248c0c-a11a-4d6b-b273-2abb5d1434d6" path="/var/lib/kubelet/pods/61248c0c-a11a-4d6b-b273-2abb5d1434d6/volumes" Nov 02 17:49:04.490200 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:49:04.490159 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:49:34.486659 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:49:34.486592 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:49:36.085328 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:36.085298 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg"] Nov 02 17:49:39.103750 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.103718 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk"] Nov 02 17:49:39.104666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.104582 5601 cpu_manager.go:414] "containerMap: removing container" podUID="61248c0c-a11a-4d6b-b273-2abb5d1434d6" containerName="istio-proxy" Nov 02 17:49:39.104666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.104602 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61248c0c-a11a-4d6b-b273-2abb5d1434d6" containerName="istio-proxy" Nov 02 17:49:39.104844 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.104712 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="61248c0c-a11a-4d6b-b273-2abb5d1434d6" containerName="istio-proxy" Nov 02 17:49:39.108530 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.108509 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.125198 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.125165 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-dd1cba-openshift-default-dockercfg-fmw9p\"" Nov 02 17:49:39.129433 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.129405 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk"] Nov 02 17:49:39.282225 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.282186 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-workload-socket\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.282536 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.282499 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlflp\" (UniqueName: \"kubernetes.io/projected/a5a70cf3-0372-41d0-a78f-c6e396962041-kube-api-access-jlflp\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.282738 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.282710 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-data\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.282930 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.282899 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-envoy\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.283109 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.283080 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-workload-certs\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.283236 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.283221 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-credential-socket\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.283431 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.283416 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a70cf3-0372-41d0-a78f-c6e396962041-istiod-ca-cert\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.283579 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.283565 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-podinfo\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.283726 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.283711 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-token\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.387666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.387501 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a70cf3-0372-41d0-a78f-c6e396962041-istiod-ca-cert\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.387666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.387577 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-podinfo\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.387666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.387638 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-token\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.387967 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.387742 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-workload-socket\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.387967 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.387794 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlflp\" (UniqueName: \"kubernetes.io/projected/a5a70cf3-0372-41d0-a78f-c6e396962041-kube-api-access-jlflp\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.387967 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.387848 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-data\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.387967 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.387910 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-envoy\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.388179 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.387988 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-workload-certs\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.388179 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.388015 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-credential-socket\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.389215 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.388734 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-workload-socket\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.389215 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.389119 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-workload-certs\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.389402 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.389374 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-data\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.389565 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.389546 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-credential-socket\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.389762 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.389737 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a70cf3-0372-41d0-a78f-c6e396962041-istiod-ca-cert\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.392061 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.392018 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-envoy\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.392263 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.392240 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-podinfo\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.398786 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.398765 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-token\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.399432 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.399218 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlflp\" (UniqueName: \"kubernetes.io/projected/a5a70cf3-0372-41d0-a78f-c6e396962041-kube-api-access-jlflp\") pod \"e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.453278 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.453245 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:39.765400 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.765364 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk"] Nov 02 17:49:39.775343 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:49:39.775307 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a70cf3_0372_41d0_a78f_c6e396962041.slice/crio-57d512d420cf626051729821dab35fb35accd8a4f3c29960a15c4961ef682213 WatchSource:0}: Error finding container 57d512d420cf626051729821dab35fb35accd8a4f3c29960a15c4961ef682213: Status 404 returned error can't find the container with id 57d512d420cf626051729821dab35fb35accd8a4f3c29960a15c4961ef682213 Nov 02 17:49:39.778239 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.778204 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:49:39.778333 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.778282 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:49:39.778383 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:39.778337 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:49:40.401363 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:40.401317 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" event={"ID":"a5a70cf3-0372-41d0-a78f-c6e396962041","Type":"ContainerStarted","Data":"0ac9b3d79618ca0a11e8e34ef33bb38a7b796d1d384537e9f7ecfea640fe00e2"} Nov 02 17:49:40.401884 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:40.401394 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" event={"ID":"a5a70cf3-0372-41d0-a78f-c6e396962041","Type":"ContainerStarted","Data":"57d512d420cf626051729821dab35fb35accd8a4f3c29960a15c4961ef682213"} Nov 02 17:49:40.426452 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:40.426232 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" podStartSLOduration=1.4262147299999999 podStartE2EDuration="1.42621473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:40.425499543 +0000 UTC m=+945.116553554" watchObservedRunningTime="2026-11-02 17:49:40.42621473 +0000 UTC m=+945.117268750" Nov 02 17:49:40.454397 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:40.454366 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:40.460379 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:40.460352 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:41.408764 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:41.408728 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:41.409701 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:41.409678 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:49:51.107747 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.107683 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" podUID="3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb" containerName="main" containerID="cri-o://076593f510bf4a6e9d432857ee8fafdf3a3895a5c4dbf4680b8b56cacdf04b39" gracePeriod=45 Nov 02 17:49:51.234244 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.234199 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" podUID="3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": dial tcp 10.134.0.88:8000: connect: connection refused" Nov 02 17:49:51.491032 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.490992 5601 generic.go:356] "Generic (PLEG): container finished" podID="3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb" containerID="076593f510bf4a6e9d432857ee8fafdf3a3895a5c4dbf4680b8b56cacdf04b39" exitCode=0 Nov 02 17:49:51.491204 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.491042 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" event={"ID":"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb","Type":"ContainerDied","Data":"076593f510bf4a6e9d432857ee8fafdf3a3895a5c4dbf4680b8b56cacdf04b39"} Nov 02 17:49:51.491204 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.491076 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" event={"ID":"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb","Type":"ContainerDied","Data":"f0a1fe003be2fe630020033d33f9a17fc0f9fca586e51c78951e7a1b4af17a14"} Nov 02 17:49:51.491204 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.491092 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0a1fe003be2fe630020033d33f9a17fc0f9fca586e51c78951e7a1b4af17a14" Nov 02 17:49:51.506066 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.506037 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:49:51.621587 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.621555 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-tmp-dir\") pod \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " Nov 02 17:49:51.621820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.621594 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-model-cache\") pod \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " Nov 02 17:49:51.621820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.621660 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-kube-api-access-sgzp7\" (UniqueName: \"kubernetes.io/projected/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-kube-api-access-sgzp7\") pod \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " Nov 02 17:49:51.621951 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.621838 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-dshm\") pod \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " Nov 02 17:49:51.621951 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.621860 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-model-cache" pod "3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb" (UID: "3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:51.621951 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.621883 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-tmp-dir" pod "3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb" (UID: "3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:51.621951 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.621911 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-tls-certs\") pod \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " Nov 02 17:49:51.622173 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.621954 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-home\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-home\") pod \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\" (UID: \"3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb\") " Nov 02 17:49:51.622272 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.622242 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-home" pod "3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb" (UID: "3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:51.622558 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.622540 5601 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-tmp-dir\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:49:51.622716 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.622560 5601 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-model-cache\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:49:51.622716 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.622569 5601 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-home\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:49:51.623949 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.623927 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-kube-api-access-sgzp7" pod "3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb" (UID: "3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb"). InnerVolumeSpecName "kube-api-access-sgzp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:51.624351 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.624330 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-tls-certs" pod "3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb" (UID: "3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:49:51.624351 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.624338 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-dshm" pod "3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb" (UID: "3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:51.724268 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.724184 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgzp7\" (UniqueName: \"kubernetes.io/projected/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-kube-api-access-sgzp7\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:49:51.724268 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.724216 5601 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-dshm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:49:51.724268 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:51.724226 5601 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb-tls-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:49:52.499288 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.499240 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg" Nov 02 17:49:52.533497 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.533469 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg"] Nov 02 17:49:52.538295 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.538272 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-da5f07e6/e2e-lifecycle-model-da5f07e6-kserve-854dc4b77-wrsdg"] Nov 02 17:49:52.623975 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.623943 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb"] Nov 02 17:49:52.624874 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.624595 5601 cpu_manager.go:414] "containerMap: removing container" podUID="3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb" containerName="main" Nov 02 17:49:52.624874 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.624721 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb" containerName="main" Nov 02 17:49:52.625056 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.624969 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb" containerName="main" Nov 02 17:49:52.628794 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.628776 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" Nov 02 17:49:52.636272 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.636253 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-dd1cba-serving-cert\"" Nov 02 17:49:52.640072 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.640054 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb"] Nov 02 17:49:52.735772 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.735738 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d301da1-7d1b-4b15-af60-61344075edb7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb\" (UID: \"3d301da1-7d1b-4b15-af60-61344075edb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" Nov 02 17:49:52.735973 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.735867 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d301da1-7d1b-4b15-af60-61344075edb7-metrics-certs\") pod \"maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb\" (UID: \"3d301da1-7d1b-4b15-af60-61344075edb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" Nov 02 17:49:52.735973 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.735890 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzvbh\" (UniqueName: \"kubernetes.io/projected/3d301da1-7d1b-4b15-af60-61344075edb7-kube-api-access-pzvbh\") pod \"maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb\" (UID: \"3d301da1-7d1b-4b15-af60-61344075edb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" Nov 02 17:49:52.736090 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.736017 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d301da1-7d1b-4b15-af60-61344075edb7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb\" (UID: \"3d301da1-7d1b-4b15-af60-61344075edb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" Nov 02 17:49:52.838051 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.837964 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d301da1-7d1b-4b15-af60-61344075edb7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb\" (UID: \"3d301da1-7d1b-4b15-af60-61344075edb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" Nov 02 17:49:52.838217 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.838089 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d301da1-7d1b-4b15-af60-61344075edb7-metrics-certs\") pod \"maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb\" (UID: \"3d301da1-7d1b-4b15-af60-61344075edb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" Nov 02 17:49:52.838217 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.838121 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzvbh\" (UniqueName: \"kubernetes.io/projected/3d301da1-7d1b-4b15-af60-61344075edb7-kube-api-access-pzvbh\") pod \"maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb\" (UID: \"3d301da1-7d1b-4b15-af60-61344075edb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" Nov 02 17:49:52.838217 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.838187 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d301da1-7d1b-4b15-af60-61344075edb7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb\" (UID: \"3d301da1-7d1b-4b15-af60-61344075edb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" Nov 02 17:49:52.838925 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.838876 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d301da1-7d1b-4b15-af60-61344075edb7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb\" (UID: \"3d301da1-7d1b-4b15-af60-61344075edb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" Nov 02 17:49:52.840845 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.840583 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d301da1-7d1b-4b15-af60-61344075edb7-metrics-certs\") pod \"maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb\" (UID: \"3d301da1-7d1b-4b15-af60-61344075edb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" Nov 02 17:49:52.841036 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.841012 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d301da1-7d1b-4b15-af60-61344075edb7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb\" (UID: \"3d301da1-7d1b-4b15-af60-61344075edb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" Nov 02 17:49:52.850032 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.850006 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzvbh\" (UniqueName: \"kubernetes.io/projected/3d301da1-7d1b-4b15-af60-61344075edb7-kube-api-access-pzvbh\") pod \"maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb\" (UID: \"3d301da1-7d1b-4b15-af60-61344075edb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" Nov 02 17:49:52.951453 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:52.951421 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" Nov 02 17:49:53.234955 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:53.234925 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb"] Nov 02 17:49:53.235752 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:49:53.235718 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d301da1_7d1b_4b15_af60_61344075edb7.slice/crio-d58c52abbc7bbdba652a99d862f29e1235feff9008ac0675196dd7e0bc3fcf07 WatchSource:0}: Error finding container d58c52abbc7bbdba652a99d862f29e1235feff9008ac0675196dd7e0bc3fcf07: Status 404 returned error can't find the container with id d58c52abbc7bbdba652a99d862f29e1235feff9008ac0675196dd7e0bc3fcf07 Nov 02 17:49:53.430097 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:53.430064 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb" path="/var/lib/kubelet/pods/3dd9a0c1-37b9-48a4-a50a-cb59e09a5ecb/volumes" Nov 02 17:49:53.507158 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:53.507072 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" event={"ID":"3d301da1-7d1b-4b15-af60-61344075edb7","Type":"ContainerStarted","Data":"d58c52abbc7bbdba652a99d862f29e1235feff9008ac0675196dd7e0bc3fcf07"} Nov 02 17:49:54.516210 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:54.516173 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" event={"ID":"3d301da1-7d1b-4b15-af60-61344075edb7","Type":"ContainerStarted","Data":"c56931c4a5e2c22dfbee7a16b0070c9a1e0de582143365447d853c872db75044"} Nov 02 17:49:54.516639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:54.516284 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" Nov 02 17:49:54.536053 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:49:54.536007 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" podStartSLOduration=1.961727985 podStartE2EDuration="2.535994518s" totalImagesPullingTime="574.266533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:54.534812097 +0000 UTC m=+959.225866136" watchObservedRunningTime="2026-11-02 17:49:54.535994518 +0000 UTC m=+959.227048530" Nov 02 17:50:00.459596 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:00.459559 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z"] Nov 02 17:50:00.465880 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:00.465858 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z" Nov 02 17:50:00.477434 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:00.475480 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z"] Nov 02 17:50:00.477434 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:00.475727 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c8832b-dockercfg-dn6px\"" Nov 02 17:50:00.531536 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:00.531508 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" Nov 02 17:50:00.564667 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:00.560830 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c8832b\"" Nov 02 17:50:00.578805 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:00.570167 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7040aee0-08fe-4e64-b18d-4d0a4f38fd8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z\" (UID: \"7040aee0-08fe-4e64-b18d-4d0a4f38fd8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z" Nov 02 17:50:00.578805 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:00.570354 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6pdm\" (UniqueName: \"kubernetes.io/projected/7040aee0-08fe-4e64-b18d-4d0a4f38fd8e-kube-api-access-h6pdm\") pod \"payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z\" (UID: \"7040aee0-08fe-4e64-b18d-4d0a4f38fd8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z" Nov 02 17:50:00.691374 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:00.691328 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7040aee0-08fe-4e64-b18d-4d0a4f38fd8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z\" (UID: \"7040aee0-08fe-4e64-b18d-4d0a4f38fd8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z" Nov 02 17:50:00.695542 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:00.693283 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7040aee0-08fe-4e64-b18d-4d0a4f38fd8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z\" (UID: \"7040aee0-08fe-4e64-b18d-4d0a4f38fd8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z" Nov 02 17:50:00.696074 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:00.696054 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6pdm\" (UniqueName: \"kubernetes.io/projected/7040aee0-08fe-4e64-b18d-4d0a4f38fd8e-kube-api-access-h6pdm\") pod \"payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z\" (UID: \"7040aee0-08fe-4e64-b18d-4d0a4f38fd8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z" Nov 02 17:50:00.782725 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:00.770386 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6pdm\" (UniqueName: \"kubernetes.io/projected/7040aee0-08fe-4e64-b18d-4d0a4f38fd8e-kube-api-access-h6pdm\") pod \"payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z\" (UID: \"7040aee0-08fe-4e64-b18d-4d0a4f38fd8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z" Nov 02 17:50:00.796846 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:00.796810 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z" Nov 02 17:50:01.084773 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:01.084743 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z"] Nov 02 17:50:01.085742 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:50:01.085718 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7040aee0_08fe_4e64_b18d_4d0a4f38fd8e.slice/crio-e710ec1031db4951e4a9baadb15ff1aab92f7ef095cc21e9005cc53d22f41cb2 WatchSource:0}: Error finding container e710ec1031db4951e4a9baadb15ff1aab92f7ef095cc21e9005cc53d22f41cb2: Status 404 returned error can't find the container with id e710ec1031db4951e4a9baadb15ff1aab92f7ef095cc21e9005cc53d22f41cb2 Nov 02 17:50:01.681627 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:01.681576 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z" event={"ID":"7040aee0-08fe-4e64-b18d-4d0a4f38fd8e","Type":"ContainerStarted","Data":"81ec3816a34ef63002c0cb09f652667d9327caf958634fe2e5d95484fad3b526"} Nov 02 17:50:01.682147 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:01.681669 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z" Nov 02 17:50:01.682147 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:01.681688 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z" event={"ID":"7040aee0-08fe-4e64-b18d-4d0a4f38fd8e","Type":"ContainerStarted","Data":"e710ec1031db4951e4a9baadb15ff1aab92f7ef095cc21e9005cc53d22f41cb2"} Nov 02 17:50:01.699681 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:01.699604 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z" podStartSLOduration=1.699590127 podStartE2EDuration="1.699590127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:01.69835959 +0000 UTC m=+966.389413603" watchObservedRunningTime="2026-11-02 17:50:01.699590127 +0000 UTC m=+966.390644138" Nov 02 17:50:04.485788 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:50:04.485752 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:50:08.790660 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:08.790080 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk"] Nov 02 17:50:08.795113 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:08.795088 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" Nov 02 17:50:08.810282 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:08.810248 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c8832b-serving-cert\"" Nov 02 17:50:08.822919 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:08.822886 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk"] Nov 02 17:50:08.854780 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:08.854742 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk\" (UID: \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" Nov 02 17:50:08.854980 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:08.854881 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk\" (UID: \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" Nov 02 17:50:08.854980 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:08.854929 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvthp\" (UniqueName: \"kubernetes.io/projected/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-kube-api-access-tvthp\") pod \"maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk\" (UID: \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" Nov 02 17:50:08.855159 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:08.855131 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-metrics-certs\") pod \"maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk\" (UID: \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" Nov 02 17:50:08.956891 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:08.956849 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvthp\" (UniqueName: \"kubernetes.io/projected/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-kube-api-access-tvthp\") pod \"maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk\" (UID: \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" Nov 02 17:50:08.957104 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:08.956989 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-metrics-certs\") pod \"maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk\" (UID: \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" Nov 02 17:50:08.957104 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:08.957078 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk\" (UID: \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" Nov 02 17:50:08.957238 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:08.957222 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk\" (UID: \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" Nov 02 17:50:08.957295 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:50:08.957242 5601 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-serving-cert: secret "maas-api-e2e-ipp-b-c8832b-serving-cert" not found Nov 02 17:50:08.957342 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:50:08.957314 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-maas-api-tls podName:e1158d50-0ccc-4cf3-b7f5-967e25c61bab nodeName:}" failed. No retries permitted until 2026-11-02 17:50:09.45729664 +0000 UTC m=+974.148350633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-maas-api-tls") pod "maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" (UID: "e1158d50-0ccc-4cf3-b7f5-967e25c61bab") : secret "maas-api-e2e-ipp-b-c8832b-serving-cert" not found Nov 02 17:50:08.957835 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:08.957811 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk\" (UID: \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" Nov 02 17:50:08.959633 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:08.959598 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-metrics-certs\") pod \"maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk\" (UID: \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" Nov 02 17:50:08.966609 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:08.966579 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvthp\" (UniqueName: \"kubernetes.io/projected/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-kube-api-access-tvthp\") pod \"maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk\" (UID: \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" Nov 02 17:50:09.479641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:09.478941 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk\" (UID: \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" Nov 02 17:50:09.515640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:09.515257 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk\" (UID: \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" Nov 02 17:50:09.739061 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:09.738959 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" Nov 02 17:50:10.030396 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:10.030305 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m"] Nov 02 17:50:10.033557 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:10.033528 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e08dec13-d8x8m"] Nov 02 17:50:10.042210 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:10.042186 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk"] Nov 02 17:50:10.043198 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:50:10.043170 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1158d50_0ccc_4cf3_b7f5_967e25c61bab.slice/crio-7f501a16149139c02365208b05344ad838031936ee20733658356b2e535aba4a WatchSource:0}: Error finding container 7f501a16149139c02365208b05344ad838031936ee20733658356b2e535aba4a: Status 404 returned error can't find the container with id 7f501a16149139c02365208b05344ad838031936ee20733658356b2e535aba4a Nov 02 17:50:10.759212 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:10.759117 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" event={"ID":"e1158d50-0ccc-4cf3-b7f5-967e25c61bab","Type":"ContainerStarted","Data":"6c0f7bd93608a2ebb2ca0bbeb4351dbbb88cf4c8e82405141a31a83afcb99353"} Nov 02 17:50:10.759212 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:10.759175 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" event={"ID":"e1158d50-0ccc-4cf3-b7f5-967e25c61bab","Type":"ContainerStarted","Data":"7f501a16149139c02365208b05344ad838031936ee20733658356b2e535aba4a"} Nov 02 17:50:10.759495 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:10.759307 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" Nov 02 17:50:11.427228 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:11.427188 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5177fbbe-15f1-4b55-84dd-1f2d2a952617" path="/var/lib/kubelet/pods/5177fbbe-15f1-4b55-84dd-1f2d2a952617/volumes" Nov 02 17:50:12.707500 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:12.707469 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z" Nov 02 17:50:12.772639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:12.740765 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" podStartSLOduration=4.305425392 podStartE2EDuration="4.740741026s" totalImagesPullingTime="435.315634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:10.780514832 +0000 UTC m=+975.471568843" watchObservedRunningTime="2026-11-02 17:50:12.740741026 +0000 UTC m=+977.431795043" Nov 02 17:50:16.799175 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:16.799147 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" Nov 02 17:50:20.917827 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:20.917731 5601 generic.go:356] "Generic (PLEG): container finished" podID="a5a70cf3-0372-41d0-a78f-c6e396962041" containerID="0ac9b3d79618ca0a11e8e34ef33bb38a7b796d1d384537e9f7ecfea640fe00e2" exitCode=0 Nov 02 17:50:20.917827 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:20.917812 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" event={"ID":"a5a70cf3-0372-41d0-a78f-c6e396962041","Type":"ContainerDied","Data":"0ac9b3d79618ca0a11e8e34ef33bb38a7b796d1d384537e9f7ecfea640fe00e2"} Nov 02 17:50:20.918299 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:20.918283 5601 scope.go:122] "RemoveContainer" containerID="0ac9b3d79618ca0a11e8e34ef33bb38a7b796d1d384537e9f7ecfea640fe00e2" Nov 02 17:50:20.920145 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:20.920100 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:50:20.920234 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:20.920200 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:50:20.920280 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:20.920249 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:50:21.926353 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:21.926314 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" event={"ID":"a5a70cf3-0372-41d0-a78f-c6e396962041","Type":"ContainerStarted","Data":"bcf8bd18b57d1979680d2cee95f2bbc77a3d92a8e900587077fe82c8a1687481"} Nov 02 17:50:22.459644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:22.456191 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:50:22.459644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:22.456334 5601 patch_prober.go:29] interesting pod/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 17:50:22.459644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:22.456390 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" podUID="a5a70cf3-0372-41d0-a78f-c6e396962041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 17:50:23.457642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:23.456700 5601 patch_prober.go:29] interesting pod/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 17:50:23.457642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:23.456758 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" podUID="a5a70cf3-0372-41d0-a78f-c6e396962041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 17:50:24.454640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:24.454047 5601 patch_prober.go:29] interesting pod/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 17:50:24.454640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:24.454125 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" podUID="a5a70cf3-0372-41d0-a78f-c6e396962041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 17:50:24.513695 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:24.513647 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z"] Nov 02 17:50:25.458859 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:25.458805 5601 patch_prober.go:29] interesting pod/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 17:50:25.459076 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:25.458931 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" podUID="a5a70cf3-0372-41d0-a78f-c6e396962041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 17:50:26.409548 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:26.409514 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:50:26.458962 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:26.458739 5601 patch_prober.go:29] interesting pod/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 17:50:26.458962 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:26.458807 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" podUID="a5a70cf3-0372-41d0-a78f-c6e396962041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 17:50:27.541376 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:27.541340 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:50:27.542561 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:27.542540 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:50:29.516415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:29.515745 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z" podUID="7040aee0-08fe-4e64-b18d-4d0a4f38fd8e" containerName="payload-pre-processing" containerID="cri-o://81ec3816a34ef63002c0cb09f652667d9327caf958634fe2e5d95484fad3b526" gracePeriod=25 Nov 02 17:50:29.519274 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:29.519240 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn"] Nov 02 17:50:29.521384 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:29.521349 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" podUID="3f62a981-07b4-4450-9013-959356a33a36" containerName="istio-proxy" containerID="cri-o://6db0d814f76aa85f6c9cef030d026ce10219c51a1c88850fed2408818a439cad" gracePeriod=30 Nov 02 17:50:29.932652 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:29.932600 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z" Nov 02 17:50:30.033986 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:30.033902 5601 generic.go:356] "Generic (PLEG): container finished" podID="7040aee0-08fe-4e64-b18d-4d0a4f38fd8e" containerID="81ec3816a34ef63002c0cb09f652667d9327caf958634fe2e5d95484fad3b526" exitCode=0 Nov 02 17:50:30.033986 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:30.033954 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z" Nov 02 17:50:30.033986 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:30.033970 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z" event={"ID":"7040aee0-08fe-4e64-b18d-4d0a4f38fd8e","Type":"ContainerDied","Data":"81ec3816a34ef63002c0cb09f652667d9327caf958634fe2e5d95484fad3b526"} Nov 02 17:50:30.034250 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:30.034012 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z" event={"ID":"7040aee0-08fe-4e64-b18d-4d0a4f38fd8e","Type":"ContainerDied","Data":"e710ec1031db4951e4a9baadb15ff1aab92f7ef095cc21e9005cc53d22f41cb2"} Nov 02 17:50:30.034250 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:30.034033 5601 scope.go:122] "RemoveContainer" containerID="81ec3816a34ef63002c0cb09f652667d9327caf958634fe2e5d95484fad3b526" Nov 02 17:50:30.038830 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:30.038800 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7040aee0-08fe-4e64-b18d-4d0a4f38fd8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7040aee0-08fe-4e64-b18d-4d0a4f38fd8e-plugins-config-volume\") pod \"7040aee0-08fe-4e64-b18d-4d0a4f38fd8e\" (UID: \"7040aee0-08fe-4e64-b18d-4d0a4f38fd8e\") " Nov 02 17:50:30.038941 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:30.038867 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7040aee0-08fe-4e64-b18d-4d0a4f38fd8e-kube-api-access-h6pdm\" (UniqueName: \"kubernetes.io/projected/7040aee0-08fe-4e64-b18d-4d0a4f38fd8e-kube-api-access-h6pdm\") pod \"7040aee0-08fe-4e64-b18d-4d0a4f38fd8e\" (UID: \"7040aee0-08fe-4e64-b18d-4d0a4f38fd8e\") " Nov 02 17:50:30.039199 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:30.039173 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7040aee0-08fe-4e64-b18d-4d0a4f38fd8e-plugins-config-volume" pod "7040aee0-08fe-4e64-b18d-4d0a4f38fd8e" (UID: "7040aee0-08fe-4e64-b18d-4d0a4f38fd8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:50:30.039501 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:30.039474 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7040aee0-08fe-4e64-b18d-4d0a4f38fd8e-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:30.041148 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:30.041120 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7040aee0-08fe-4e64-b18d-4d0a4f38fd8e-kube-api-access-h6pdm" pod "7040aee0-08fe-4e64-b18d-4d0a4f38fd8e" (UID: "7040aee0-08fe-4e64-b18d-4d0a4f38fd8e"). InnerVolumeSpecName "kube-api-access-h6pdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:30.045685 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:30.045561 5601 scope.go:122] "RemoveContainer" containerID="81ec3816a34ef63002c0cb09f652667d9327caf958634fe2e5d95484fad3b526" Nov 02 17:50:30.046219 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:50:30.046179 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ec3816a34ef63002c0cb09f652667d9327caf958634fe2e5d95484fad3b526\": container with ID starting with 81ec3816a34ef63002c0cb09f652667d9327caf958634fe2e5d95484fad3b526 not found: ID does not exist" containerID="81ec3816a34ef63002c0cb09f652667d9327caf958634fe2e5d95484fad3b526" Nov 02 17:50:30.046331 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:30.046218 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ec3816a34ef63002c0cb09f652667d9327caf958634fe2e5d95484fad3b526"} err="failed to get container status \"81ec3816a34ef63002c0cb09f652667d9327caf958634fe2e5d95484fad3b526\": rpc error: code = NotFound desc = could not find container \"81ec3816a34ef63002c0cb09f652667d9327caf958634fe2e5d95484fad3b526\": container with ID starting with 81ec3816a34ef63002c0cb09f652667d9327caf958634fe2e5d95484fad3b526 not found: ID does not exist" Nov 02 17:50:30.140908 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:30.140874 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6pdm\" (UniqueName: \"kubernetes.io/projected/7040aee0-08fe-4e64-b18d-4d0a4f38fd8e-kube-api-access-h6pdm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:30.360308 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:30.360272 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z"] Nov 02 17:50:30.363281 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:30.363254 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-5c5f6c56c5-rkb6z"] Nov 02 17:50:31.429932 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:31.429897 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7040aee0-08fe-4e64-b18d-4d0a4f38fd8e" path="/var/lib/kubelet/pods/7040aee0-08fe-4e64-b18d-4d0a4f38fd8e/volumes" Nov 02 17:50:34.489799 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:50:34.489758 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:50:34.840528 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.840497 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:50:34.916389 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.916353 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-istio-data\") pod \"3f62a981-07b4-4450-9013-959356a33a36\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " Nov 02 17:50:34.916569 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.916409 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-workload-certs\") pod \"3f62a981-07b4-4450-9013-959356a33a36\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " Nov 02 17:50:34.916569 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.916459 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f62a981-07b4-4450-9013-959356a33a36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f62a981-07b4-4450-9013-959356a33a36-istiod-ca-cert\") pod \"3f62a981-07b4-4450-9013-959356a33a36\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " Nov 02 17:50:34.916569 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.916554 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-workload-socket\") pod \"3f62a981-07b4-4450-9013-959356a33a36\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " Nov 02 17:50:34.916785 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.916584 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f62a981-07b4-4450-9013-959356a33a36-kube-api-access-l4q4c\" (UniqueName: \"kubernetes.io/projected/3f62a981-07b4-4450-9013-959356a33a36-kube-api-access-l4q4c\") pod \"3f62a981-07b4-4450-9013-959356a33a36\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " Nov 02 17:50:34.916785 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.916693 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-istio-data" pod "3f62a981-07b4-4450-9013-959356a33a36" (UID: "3f62a981-07b4-4450-9013-959356a33a36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:34.916785 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.916703 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f62a981-07b4-4450-9013-959356a33a36-istio-token\" (UniqueName: \"kubernetes.io/projected/3f62a981-07b4-4450-9013-959356a33a36-istio-token\") pod \"3f62a981-07b4-4450-9013-959356a33a36\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " Nov 02 17:50:34.916785 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.916754 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-istio-envoy\") pod \"3f62a981-07b4-4450-9013-959356a33a36\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " Nov 02 17:50:34.916994 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.916838 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f62a981-07b4-4450-9013-959356a33a36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f62a981-07b4-4450-9013-959356a33a36-istio-podinfo\") pod \"3f62a981-07b4-4450-9013-959356a33a36\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " Nov 02 17:50:34.916994 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.916901 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-workload-socket" pod "3f62a981-07b4-4450-9013-959356a33a36" (UID: "3f62a981-07b4-4450-9013-959356a33a36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:34.916994 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.916927 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-credential-socket\") pod \"3f62a981-07b4-4450-9013-959356a33a36\" (UID: \"3f62a981-07b4-4450-9013-959356a33a36\") " Nov 02 17:50:34.916994 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.916932 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f62a981-07b4-4450-9013-959356a33a36-istiod-ca-cert" pod "3f62a981-07b4-4450-9013-959356a33a36" (UID: "3f62a981-07b4-4450-9013-959356a33a36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:50:34.917199 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.917116 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-workload-certs" pod "3f62a981-07b4-4450-9013-959356a33a36" (UID: "3f62a981-07b4-4450-9013-959356a33a36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:34.917320 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.917296 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-credential-socket" pod "3f62a981-07b4-4450-9013-959356a33a36" (UID: "3f62a981-07b4-4450-9013-959356a33a36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:34.917476 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.917457 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:34.917476 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.917476 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:34.917683 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.917491 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f62a981-07b4-4450-9013-959356a33a36-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:34.917683 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.917504 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:34.917683 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.917513 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:34.919018 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.918998 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f62a981-07b4-4450-9013-959356a33a36-istio-podinfo" pod "3f62a981-07b4-4450-9013-959356a33a36" (UID: "3f62a981-07b4-4450-9013-959356a33a36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:50:34.919100 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.919044 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f62a981-07b4-4450-9013-959356a33a36-istio-token" pod "3f62a981-07b4-4450-9013-959356a33a36" (UID: "3f62a981-07b4-4450-9013-959356a33a36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:34.919264 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.919247 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-istio-envoy" pod "3f62a981-07b4-4450-9013-959356a33a36" (UID: "3f62a981-07b4-4450-9013-959356a33a36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:34.919308 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:34.919268 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f62a981-07b4-4450-9013-959356a33a36-kube-api-access-l4q4c" pod "3f62a981-07b4-4450-9013-959356a33a36" (UID: "3f62a981-07b4-4450-9013-959356a33a36"). InnerVolumeSpecName "kube-api-access-l4q4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:35.019277 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:35.019249 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4q4c\" (UniqueName: \"kubernetes.io/projected/3f62a981-07b4-4450-9013-959356a33a36-kube-api-access-l4q4c\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:35.019277 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:35.019278 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f62a981-07b4-4450-9013-959356a33a36-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:35.019462 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:35.019289 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f62a981-07b4-4450-9013-959356a33a36-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:35.019462 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:35.019298 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f62a981-07b4-4450-9013-959356a33a36-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:50:35.080273 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:35.080184 5601 generic.go:356] "Generic (PLEG): container finished" podID="3f62a981-07b4-4450-9013-959356a33a36" containerID="6db0d814f76aa85f6c9cef030d026ce10219c51a1c88850fed2408818a439cad" exitCode=0 Nov 02 17:50:35.080273 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:35.080237 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" Nov 02 17:50:35.080477 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:35.080266 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" event={"ID":"3f62a981-07b4-4450-9013-959356a33a36","Type":"ContainerDied","Data":"6db0d814f76aa85f6c9cef030d026ce10219c51a1c88850fed2408818a439cad"} Nov 02 17:50:35.080477 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:35.080321 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn" event={"ID":"3f62a981-07b4-4450-9013-959356a33a36","Type":"ContainerDied","Data":"4c31802a1680a46a73345036242e30c50665c9bfd841efecaf16121bc0b761ba"} Nov 02 17:50:35.080477 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:35.080342 5601 scope.go:122] "RemoveContainer" containerID="6db0d814f76aa85f6c9cef030d026ce10219c51a1c88850fed2408818a439cad" Nov 02 17:50:35.092977 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:35.092955 5601 scope.go:122] "RemoveContainer" containerID="3c084d3b72fd58de7e76b9b67193e713b508ba8493d39cc76bc9329cedbf059f" Nov 02 17:50:35.101869 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:35.101849 5601 scope.go:122] "RemoveContainer" containerID="6db0d814f76aa85f6c9cef030d026ce10219c51a1c88850fed2408818a439cad" Nov 02 17:50:35.102130 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:50:35.102107 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6db0d814f76aa85f6c9cef030d026ce10219c51a1c88850fed2408818a439cad\": container with ID starting with 6db0d814f76aa85f6c9cef030d026ce10219c51a1c88850fed2408818a439cad not found: ID does not exist" containerID="6db0d814f76aa85f6c9cef030d026ce10219c51a1c88850fed2408818a439cad" Nov 02 17:50:35.102209 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:35.102143 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6db0d814f76aa85f6c9cef030d026ce10219c51a1c88850fed2408818a439cad"} err="failed to get container status \"6db0d814f76aa85f6c9cef030d026ce10219c51a1c88850fed2408818a439cad\": rpc error: code = NotFound desc = could not find container \"6db0d814f76aa85f6c9cef030d026ce10219c51a1c88850fed2408818a439cad\": container with ID starting with 6db0d814f76aa85f6c9cef030d026ce10219c51a1c88850fed2408818a439cad not found: ID does not exist" Nov 02 17:50:35.102209 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:35.102176 5601 scope.go:122] "RemoveContainer" containerID="3c084d3b72fd58de7e76b9b67193e713b508ba8493d39cc76bc9329cedbf059f" Nov 02 17:50:35.102493 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:50:35.102471 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c084d3b72fd58de7e76b9b67193e713b508ba8493d39cc76bc9329cedbf059f\": container with ID starting with 3c084d3b72fd58de7e76b9b67193e713b508ba8493d39cc76bc9329cedbf059f not found: ID does not exist" containerID="3c084d3b72fd58de7e76b9b67193e713b508ba8493d39cc76bc9329cedbf059f" Nov 02 17:50:35.102585 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:35.102498 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c084d3b72fd58de7e76b9b67193e713b508ba8493d39cc76bc9329cedbf059f"} err="failed to get container status \"3c084d3b72fd58de7e76b9b67193e713b508ba8493d39cc76bc9329cedbf059f\": rpc error: code = NotFound desc = could not find container \"3c084d3b72fd58de7e76b9b67193e713b508ba8493d39cc76bc9329cedbf059f\": container with ID starting with 3c084d3b72fd58de7e76b9b67193e713b508ba8493d39cc76bc9329cedbf059f not found: ID does not exist" Nov 02 17:50:35.114738 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:35.114706 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn"] Nov 02 17:50:35.118029 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:35.118002 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-mtnhn"] Nov 02 17:50:35.427987 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:35.427955 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f62a981-07b4-4450-9013-959356a33a36" path="/var/lib/kubelet/pods/3f62a981-07b4-4450-9013-959356a33a36/volumes" Nov 02 17:50:44.400741 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.400657 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt"] Nov 02 17:50:44.401341 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.401254 5601 cpu_manager.go:414] "containerMap: removing container" podUID="7040aee0-08fe-4e64-b18d-4d0a4f38fd8e" containerName="payload-pre-processing" Nov 02 17:50:44.401341 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.401268 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7040aee0-08fe-4e64-b18d-4d0a4f38fd8e" containerName="payload-pre-processing" Nov 02 17:50:44.401341 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.401298 5601 cpu_manager.go:414] "containerMap: removing container" podUID="3f62a981-07b4-4450-9013-959356a33a36" containerName="istio-proxy" Nov 02 17:50:44.401341 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.401305 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f62a981-07b4-4450-9013-959356a33a36" containerName="istio-proxy" Nov 02 17:50:44.401341 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.401325 5601 cpu_manager.go:414] "containerMap: removing container" podUID="3f62a981-07b4-4450-9013-959356a33a36" containerName="istio-proxy" Nov 02 17:50:44.401341 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.401330 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f62a981-07b4-4450-9013-959356a33a36" containerName="istio-proxy" Nov 02 17:50:44.401697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.401418 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f62a981-07b4-4450-9013-959356a33a36" containerName="istio-proxy" Nov 02 17:50:44.401697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.401430 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="7040aee0-08fe-4e64-b18d-4d0a4f38fd8e" containerName="payload-pre-processing" Nov 02 17:50:44.404848 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.404829 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" Nov 02 17:50:44.409949 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.409929 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-cd2738-serving-cert\"" Nov 02 17:50:44.414541 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.414513 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt"] Nov 02 17:50:44.463540 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.463510 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54a717b2-8169-498d-90e3-d8499f9af4de-openshift-service-ca\") pod \"maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt\" (UID: \"54a717b2-8169-498d-90e3-d8499f9af4de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" Nov 02 17:50:44.463743 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.463584 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gcwc\" (UniqueName: \"kubernetes.io/projected/54a717b2-8169-498d-90e3-d8499f9af4de-kube-api-access-4gcwc\") pod \"maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt\" (UID: \"54a717b2-8169-498d-90e3-d8499f9af4de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" Nov 02 17:50:44.463828 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.463734 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54a717b2-8169-498d-90e3-d8499f9af4de-metrics-certs\") pod \"maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt\" (UID: \"54a717b2-8169-498d-90e3-d8499f9af4de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" Nov 02 17:50:44.463828 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.463805 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54a717b2-8169-498d-90e3-d8499f9af4de-maas-api-tls\") pod \"maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt\" (UID: \"54a717b2-8169-498d-90e3-d8499f9af4de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" Nov 02 17:50:44.565358 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.565318 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gcwc\" (UniqueName: \"kubernetes.io/projected/54a717b2-8169-498d-90e3-d8499f9af4de-kube-api-access-4gcwc\") pod \"maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt\" (UID: \"54a717b2-8169-498d-90e3-d8499f9af4de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" Nov 02 17:50:44.565577 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.565391 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54a717b2-8169-498d-90e3-d8499f9af4de-metrics-certs\") pod \"maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt\" (UID: \"54a717b2-8169-498d-90e3-d8499f9af4de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" Nov 02 17:50:44.565577 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.565468 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54a717b2-8169-498d-90e3-d8499f9af4de-maas-api-tls\") pod \"maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt\" (UID: \"54a717b2-8169-498d-90e3-d8499f9af4de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" Nov 02 17:50:44.566576 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.566547 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54a717b2-8169-498d-90e3-d8499f9af4de-openshift-service-ca\") pod \"maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt\" (UID: \"54a717b2-8169-498d-90e3-d8499f9af4de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" Nov 02 17:50:44.566849 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:50:44.566826 5601 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-serving-cert: secret "maas-api-e2e-api-a-cd2738-serving-cert" not found Nov 02 17:50:44.567063 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:50:44.567041 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54a717b2-8169-498d-90e3-d8499f9af4de-maas-api-tls podName:54a717b2-8169-498d-90e3-d8499f9af4de nodeName:}" failed. No retries permitted until 2026-11-02 17:50:45.06702312 +0000 UTC m=+1009.758077126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/54a717b2-8169-498d-90e3-d8499f9af4de-maas-api-tls") pod "maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" (UID: "54a717b2-8169-498d-90e3-d8499f9af4de") : secret "maas-api-e2e-api-a-cd2738-serving-cert" not found Nov 02 17:50:44.567498 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.567475 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54a717b2-8169-498d-90e3-d8499f9af4de-openshift-service-ca\") pod \"maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt\" (UID: \"54a717b2-8169-498d-90e3-d8499f9af4de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" Nov 02 17:50:44.569391 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.569370 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54a717b2-8169-498d-90e3-d8499f9af4de-metrics-certs\") pod \"maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt\" (UID: \"54a717b2-8169-498d-90e3-d8499f9af4de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" Nov 02 17:50:44.578004 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:44.577981 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gcwc\" (UniqueName: \"kubernetes.io/projected/54a717b2-8169-498d-90e3-d8499f9af4de-kube-api-access-4gcwc\") pod \"maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt\" (UID: \"54a717b2-8169-498d-90e3-d8499f9af4de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" Nov 02 17:50:45.082020 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:45.081971 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54a717b2-8169-498d-90e3-d8499f9af4de-maas-api-tls\") pod \"maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt\" (UID: \"54a717b2-8169-498d-90e3-d8499f9af4de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" Nov 02 17:50:45.085747 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:45.085718 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54a717b2-8169-498d-90e3-d8499f9af4de-maas-api-tls\") pod \"maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt\" (UID: \"54a717b2-8169-498d-90e3-d8499f9af4de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" Nov 02 17:50:45.323143 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:45.323114 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" Nov 02 17:50:45.617650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:45.617602 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt"] Nov 02 17:50:45.618478 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:50:45.618445 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a717b2_8169_498d_90e3_d8499f9af4de.slice/crio-c48127e5e0b326c4cf399b12d885a75bebbda413894aa5246b91e0d01a39a173 WatchSource:0}: Error finding container c48127e5e0b326c4cf399b12d885a75bebbda413894aa5246b91e0d01a39a173: Status 404 returned error can't find the container with id c48127e5e0b326c4cf399b12d885a75bebbda413894aa5246b91e0d01a39a173 Nov 02 17:50:46.161820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:46.161782 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" event={"ID":"54a717b2-8169-498d-90e3-d8499f9af4de","Type":"ContainerStarted","Data":"c48127e5e0b326c4cf399b12d885a75bebbda413894aa5246b91e0d01a39a173"} Nov 02 17:50:47.170933 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:47.170894 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" event={"ID":"54a717b2-8169-498d-90e3-d8499f9af4de","Type":"ContainerStarted","Data":"b120b783d47864720ae7d90c8889586cd342b21aca969d29967019d54baf2127"} Nov 02 17:50:47.171411 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:47.171171 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" Nov 02 17:50:47.194808 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:47.194750 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" podStartSLOduration=2.617151576 podStartE2EDuration="3.194737472s" totalImagesPullingTime="577.585896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:47.191010523 +0000 UTC m=+1011.882064536" watchObservedRunningTime="2026-11-02 17:50:47.194737472 +0000 UTC m=+1011.885791484" Nov 02 17:50:53.184067 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:50:53.184034 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" Nov 02 17:51:04.484663 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:51:04.484600 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:51:10.346167 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:10.346122 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb"] Nov 02 17:51:10.346952 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:10.346900 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" podUID="3d301da1-7d1b-4b15-af60-61344075edb7" containerName="maas-api" containerID="cri-o://c56931c4a5e2c22dfbee7a16b0070c9a1e0de582143365447d853c872db75044" gracePeriod=30 Nov 02 17:51:10.526794 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:10.526753 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" podUID="3d301da1-7d1b-4b15-af60-61344075edb7" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.93:8443/health\": dial tcp 10.134.0.93:8443: connect: connection refused" Nov 02 17:51:10.815865 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:10.813811 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" Nov 02 17:51:10.908330 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:10.908284 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d301da1-7d1b-4b15-af60-61344075edb7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d301da1-7d1b-4b15-af60-61344075edb7-maas-api-tls\") pod \"3d301da1-7d1b-4b15-af60-61344075edb7\" (UID: \"3d301da1-7d1b-4b15-af60-61344075edb7\") " Nov 02 17:51:10.908529 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:10.908431 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d301da1-7d1b-4b15-af60-61344075edb7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d301da1-7d1b-4b15-af60-61344075edb7-metrics-certs\") pod \"3d301da1-7d1b-4b15-af60-61344075edb7\" (UID: \"3d301da1-7d1b-4b15-af60-61344075edb7\") " Nov 02 17:51:10.910728 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:10.910691 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d301da1-7d1b-4b15-af60-61344075edb7-maas-api-tls" pod "3d301da1-7d1b-4b15-af60-61344075edb7" (UID: "3d301da1-7d1b-4b15-af60-61344075edb7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:10.910908 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:10.910884 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d301da1-7d1b-4b15-af60-61344075edb7-metrics-certs" pod "3d301da1-7d1b-4b15-af60-61344075edb7" (UID: "3d301da1-7d1b-4b15-af60-61344075edb7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:11.010169 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:11.010086 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d301da1-7d1b-4b15-af60-61344075edb7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d301da1-7d1b-4b15-af60-61344075edb7-openshift-service-ca\") pod \"3d301da1-7d1b-4b15-af60-61344075edb7\" (UID: \"3d301da1-7d1b-4b15-af60-61344075edb7\") " Nov 02 17:51:11.010169 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:11.010128 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d301da1-7d1b-4b15-af60-61344075edb7-kube-api-access-pzvbh\" (UniqueName: \"kubernetes.io/projected/3d301da1-7d1b-4b15-af60-61344075edb7-kube-api-access-pzvbh\") pod \"3d301da1-7d1b-4b15-af60-61344075edb7\" (UID: \"3d301da1-7d1b-4b15-af60-61344075edb7\") " Nov 02 17:51:11.010459 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:11.010435 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d301da1-7d1b-4b15-af60-61344075edb7-openshift-service-ca" pod "3d301da1-7d1b-4b15-af60-61344075edb7" (UID: "3d301da1-7d1b-4b15-af60-61344075edb7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:11.010723 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:11.010705 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d301da1-7d1b-4b15-af60-61344075edb7-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:11.010801 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:11.010728 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d301da1-7d1b-4b15-af60-61344075edb7-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:11.010801 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:11.010743 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d301da1-7d1b-4b15-af60-61344075edb7-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:11.012303 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:11.012281 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d301da1-7d1b-4b15-af60-61344075edb7-kube-api-access-pzvbh" pod "3d301da1-7d1b-4b15-af60-61344075edb7" (UID: "3d301da1-7d1b-4b15-af60-61344075edb7"). InnerVolumeSpecName "kube-api-access-pzvbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:11.112564 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:11.112531 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzvbh\" (UniqueName: \"kubernetes.io/projected/3d301da1-7d1b-4b15-af60-61344075edb7-kube-api-access-pzvbh\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:11.364136 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:11.364051 5601 generic.go:356] "Generic (PLEG): container finished" podID="3d301da1-7d1b-4b15-af60-61344075edb7" containerID="c56931c4a5e2c22dfbee7a16b0070c9a1e0de582143365447d853c872db75044" exitCode=0 Nov 02 17:51:11.364136 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:11.364103 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" Nov 02 17:51:11.364741 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:11.364145 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" event={"ID":"3d301da1-7d1b-4b15-af60-61344075edb7","Type":"ContainerDied","Data":"c56931c4a5e2c22dfbee7a16b0070c9a1e0de582143365447d853c872db75044"} Nov 02 17:51:11.364741 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:11.364218 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb" event={"ID":"3d301da1-7d1b-4b15-af60-61344075edb7","Type":"ContainerDied","Data":"d58c52abbc7bbdba652a99d862f29e1235feff9008ac0675196dd7e0bc3fcf07"} Nov 02 17:51:11.364741 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:11.364252 5601 scope.go:122] "RemoveContainer" containerID="c56931c4a5e2c22dfbee7a16b0070c9a1e0de582143365447d853c872db75044" Nov 02 17:51:11.381873 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:11.381856 5601 scope.go:122] "RemoveContainer" containerID="c56931c4a5e2c22dfbee7a16b0070c9a1e0de582143365447d853c872db75044" Nov 02 17:51:11.382185 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:51:11.382160 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c56931c4a5e2c22dfbee7a16b0070c9a1e0de582143365447d853c872db75044\": container with ID starting with c56931c4a5e2c22dfbee7a16b0070c9a1e0de582143365447d853c872db75044 not found: ID does not exist" containerID="c56931c4a5e2c22dfbee7a16b0070c9a1e0de582143365447d853c872db75044" Nov 02 17:51:11.382250 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:11.382204 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c56931c4a5e2c22dfbee7a16b0070c9a1e0de582143365447d853c872db75044"} err="failed to get container status \"c56931c4a5e2c22dfbee7a16b0070c9a1e0de582143365447d853c872db75044\": rpc error: code = NotFound desc = could not find container \"c56931c4a5e2c22dfbee7a16b0070c9a1e0de582143365447d853c872db75044\": container with ID starting with c56931c4a5e2c22dfbee7a16b0070c9a1e0de582143365447d853c872db75044 not found: ID does not exist" Nov 02 17:51:11.395825 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:11.395798 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb"] Nov 02 17:51:11.400667 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:11.400644 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dd1cba-55f78b5df6-rzrhb"] Nov 02 17:51:11.427973 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:11.427945 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d301da1-7d1b-4b15-af60-61344075edb7" path="/var/lib/kubelet/pods/3d301da1-7d1b-4b15-af60-61344075edb7/volumes" Nov 02 17:51:16.551730 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:16.551693 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk"] Nov 02 17:51:16.552932 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:16.551943 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" podUID="e1158d50-0ccc-4cf3-b7f5-967e25c61bab" containerName="maas-api" containerID="cri-o://6c0f7bd93608a2ebb2ca0bbeb4351dbbb88cf4c8e82405141a31a83afcb99353" gracePeriod=30 Nov 02 17:51:16.987642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:16.981394 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" Nov 02 17:51:17.039457 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.039423 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-metrics-certs\") pod \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\" (UID: \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\") " Nov 02 17:51:17.039769 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.039468 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-openshift-service-ca\") pod \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\" (UID: \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\") " Nov 02 17:51:17.039769 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.039516 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-kube-api-access-tvthp\" (UniqueName: \"kubernetes.io/projected/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-kube-api-access-tvthp\") pod \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\" (UID: \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\") " Nov 02 17:51:17.039769 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.039676 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-maas-api-tls\") pod \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\" (UID: \"e1158d50-0ccc-4cf3-b7f5-967e25c61bab\") " Nov 02 17:51:17.039935 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.039830 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-openshift-service-ca" pod "e1158d50-0ccc-4cf3-b7f5-967e25c61bab" (UID: "e1158d50-0ccc-4cf3-b7f5-967e25c61bab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:17.040238 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.040214 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:17.045241 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.044801 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-metrics-certs" pod "e1158d50-0ccc-4cf3-b7f5-967e25c61bab" (UID: "e1158d50-0ccc-4cf3-b7f5-967e25c61bab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:17.045241 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.044878 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-maas-api-tls" pod "e1158d50-0ccc-4cf3-b7f5-967e25c61bab" (UID: "e1158d50-0ccc-4cf3-b7f5-967e25c61bab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:17.048772 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.048739 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-kube-api-access-tvthp" pod "e1158d50-0ccc-4cf3-b7f5-967e25c61bab" (UID: "e1158d50-0ccc-4cf3-b7f5-967e25c61bab"). InnerVolumeSpecName "kube-api-access-tvthp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:17.141682 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.141637 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvthp\" (UniqueName: \"kubernetes.io/projected/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-kube-api-access-tvthp\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:17.141682 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.141678 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:17.141966 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.141693 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1158d50-0ccc-4cf3-b7f5-967e25c61bab-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:17.409006 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.408967 5601 generic.go:356] "Generic (PLEG): container finished" podID="e1158d50-0ccc-4cf3-b7f5-967e25c61bab" containerID="6c0f7bd93608a2ebb2ca0bbeb4351dbbb88cf4c8e82405141a31a83afcb99353" exitCode=0 Nov 02 17:51:17.409196 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.409014 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" Nov 02 17:51:17.409196 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.409045 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" event={"ID":"e1158d50-0ccc-4cf3-b7f5-967e25c61bab","Type":"ContainerDied","Data":"6c0f7bd93608a2ebb2ca0bbeb4351dbbb88cf4c8e82405141a31a83afcb99353"} Nov 02 17:51:17.409196 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.409093 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" event={"ID":"e1158d50-0ccc-4cf3-b7f5-967e25c61bab","Type":"ContainerDied","Data":"7f501a16149139c02365208b05344ad838031936ee20733658356b2e535aba4a"} Nov 02 17:51:17.409196 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.409115 5601 scope.go:122] "RemoveContainer" containerID="6c0f7bd93608a2ebb2ca0bbeb4351dbbb88cf4c8e82405141a31a83afcb99353" Nov 02 17:51:17.419672 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.419648 5601 scope.go:122] "RemoveContainer" containerID="6c0f7bd93608a2ebb2ca0bbeb4351dbbb88cf4c8e82405141a31a83afcb99353" Nov 02 17:51:17.419985 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:51:17.419964 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0f7bd93608a2ebb2ca0bbeb4351dbbb88cf4c8e82405141a31a83afcb99353\": container with ID starting with 6c0f7bd93608a2ebb2ca0bbeb4351dbbb88cf4c8e82405141a31a83afcb99353 not found: ID does not exist" containerID="6c0f7bd93608a2ebb2ca0bbeb4351dbbb88cf4c8e82405141a31a83afcb99353" Nov 02 17:51:17.420065 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.419991 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0f7bd93608a2ebb2ca0bbeb4351dbbb88cf4c8e82405141a31a83afcb99353"} err="failed to get container status \"6c0f7bd93608a2ebb2ca0bbeb4351dbbb88cf4c8e82405141a31a83afcb99353\": rpc error: code = NotFound desc = could not find container \"6c0f7bd93608a2ebb2ca0bbeb4351dbbb88cf4c8e82405141a31a83afcb99353\": container with ID starting with 6c0f7bd93608a2ebb2ca0bbeb4351dbbb88cf4c8e82405141a31a83afcb99353 not found: ID does not exist" Nov 02 17:51:17.463520 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.463481 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk"] Nov 02 17:51:17.465793 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:17.465758 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk"] Nov 02 17:51:18.785328 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:18.785295 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk"] Nov 02 17:51:18.785838 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:18.785549 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" podUID="a5a70cf3-0372-41d0-a78f-c6e396962041" containerName="istio-proxy" containerID="cri-o://bcf8bd18b57d1979680d2cee95f2bbc77a3d92a8e900587077fe82c8a1687481" gracePeriod=30 Nov 02 17:51:19.426952 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:19.426920 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1158d50-0ccc-4cf3-b7f5-967e25c61bab" path="/var/lib/kubelet/pods/e1158d50-0ccc-4cf3-b7f5-967e25c61bab/volumes" Nov 02 17:51:19.794336 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:19.794240 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c8832b-77bdbdf85f-lkwtk" podUID="e1158d50-0ccc-4cf3-b7f5-967e25c61bab" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.95:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:51:24.056140 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.056071 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:51:24.159084 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.159043 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-envoy\") pod \"a5a70cf3-0372-41d0-a78f-c6e396962041\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " Nov 02 17:51:24.159275 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.159145 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5a70cf3-0372-41d0-a78f-c6e396962041-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a70cf3-0372-41d0-a78f-c6e396962041-istiod-ca-cert\") pod \"a5a70cf3-0372-41d0-a78f-c6e396962041\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " Nov 02 17:51:24.159275 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.159188 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-credential-socket\") pod \"a5a70cf3-0372-41d0-a78f-c6e396962041\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " Nov 02 17:51:24.159275 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.159250 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-data\") pod \"a5a70cf3-0372-41d0-a78f-c6e396962041\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " Nov 02 17:51:24.159442 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.159286 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-workload-certs\") pod \"a5a70cf3-0372-41d0-a78f-c6e396962041\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " Nov 02 17:51:24.159442 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.159325 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a70cf3-0372-41d0-a78f-c6e396962041-kube-api-access-jlflp\" (UniqueName: \"kubernetes.io/projected/a5a70cf3-0372-41d0-a78f-c6e396962041-kube-api-access-jlflp\") pod \"a5a70cf3-0372-41d0-a78f-c6e396962041\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " Nov 02 17:51:24.159442 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.159365 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-podinfo\") pod \"a5a70cf3-0372-41d0-a78f-c6e396962041\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " Nov 02 17:51:24.159442 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.159427 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-workload-socket\") pod \"a5a70cf3-0372-41d0-a78f-c6e396962041\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " Nov 02 17:51:24.159588 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.159553 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-token\" (UniqueName: \"kubernetes.io/projected/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-token\") pod \"a5a70cf3-0372-41d0-a78f-c6e396962041\" (UID: \"a5a70cf3-0372-41d0-a78f-c6e396962041\") " Nov 02 17:51:24.161399 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.161310 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a70cf3-0372-41d0-a78f-c6e396962041-istiod-ca-cert" pod "a5a70cf3-0372-41d0-a78f-c6e396962041" (UID: "a5a70cf3-0372-41d0-a78f-c6e396962041"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:24.161939 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.161599 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-credential-socket" pod "a5a70cf3-0372-41d0-a78f-c6e396962041" (UID: "a5a70cf3-0372-41d0-a78f-c6e396962041"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:24.161939 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.161827 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-workload-certs" pod "a5a70cf3-0372-41d0-a78f-c6e396962041" (UID: "a5a70cf3-0372-41d0-a78f-c6e396962041"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:24.161939 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.161853 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-workload-socket" pod "a5a70cf3-0372-41d0-a78f-c6e396962041" (UID: "a5a70cf3-0372-41d0-a78f-c6e396962041"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:24.162402 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.162165 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-data" pod "a5a70cf3-0372-41d0-a78f-c6e396962041" (UID: "a5a70cf3-0372-41d0-a78f-c6e396962041"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:24.163510 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.163462 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-envoy" pod "a5a70cf3-0372-41d0-a78f-c6e396962041" (UID: "a5a70cf3-0372-41d0-a78f-c6e396962041"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:24.164603 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.164581 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-podinfo" pod "a5a70cf3-0372-41d0-a78f-c6e396962041" (UID: "a5a70cf3-0372-41d0-a78f-c6e396962041"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:51:24.164828 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.164804 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-token" pod "a5a70cf3-0372-41d0-a78f-c6e396962041" (UID: "a5a70cf3-0372-41d0-a78f-c6e396962041"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:24.164914 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.164820 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a70cf3-0372-41d0-a78f-c6e396962041-kube-api-access-jlflp" pod "a5a70cf3-0372-41d0-a78f-c6e396962041" (UID: "a5a70cf3-0372-41d0-a78f-c6e396962041"). InnerVolumeSpecName "kube-api-access-jlflp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:24.261092 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.261054 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:24.261092 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.261089 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:24.261332 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.261101 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a70cf3-0372-41d0-a78f-c6e396962041-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:24.261332 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.261115 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:24.261332 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.261127 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:24.261332 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.261138 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:24.261332 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.261151 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlflp\" (UniqueName: \"kubernetes.io/projected/a5a70cf3-0372-41d0-a78f-c6e396962041-kube-api-access-jlflp\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:24.261332 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.261164 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a70cf3-0372-41d0-a78f-c6e396962041-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:24.261332 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.261176 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a70cf3-0372-41d0-a78f-c6e396962041-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:24.485537 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.485502 5601 generic.go:356] "Generic (PLEG): container finished" podID="a5a70cf3-0372-41d0-a78f-c6e396962041" containerID="bcf8bd18b57d1979680d2cee95f2bbc77a3d92a8e900587077fe82c8a1687481" exitCode=0 Nov 02 17:51:24.485754 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.485559 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" Nov 02 17:51:24.485754 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.485580 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" event={"ID":"a5a70cf3-0372-41d0-a78f-c6e396962041","Type":"ContainerDied","Data":"bcf8bd18b57d1979680d2cee95f2bbc77a3d92a8e900587077fe82c8a1687481"} Nov 02 17:51:24.485754 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.485650 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk" event={"ID":"a5a70cf3-0372-41d0-a78f-c6e396962041","Type":"ContainerDied","Data":"57d512d420cf626051729821dab35fb35accd8a4f3c29960a15c4961ef682213"} Nov 02 17:51:24.485754 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.485682 5601 scope.go:122] "RemoveContainer" containerID="bcf8bd18b57d1979680d2cee95f2bbc77a3d92a8e900587077fe82c8a1687481" Nov 02 17:51:24.500757 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.500729 5601 scope.go:122] "RemoveContainer" containerID="0ac9b3d79618ca0a11e8e34ef33bb38a7b796d1d384537e9f7ecfea640fe00e2" Nov 02 17:51:24.511854 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.511831 5601 scope.go:122] "RemoveContainer" containerID="bcf8bd18b57d1979680d2cee95f2bbc77a3d92a8e900587077fe82c8a1687481" Nov 02 17:51:24.512145 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:51:24.512118 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf8bd18b57d1979680d2cee95f2bbc77a3d92a8e900587077fe82c8a1687481\": container with ID starting with bcf8bd18b57d1979680d2cee95f2bbc77a3d92a8e900587077fe82c8a1687481 not found: ID does not exist" containerID="bcf8bd18b57d1979680d2cee95f2bbc77a3d92a8e900587077fe82c8a1687481" Nov 02 17:51:24.512201 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.512158 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf8bd18b57d1979680d2cee95f2bbc77a3d92a8e900587077fe82c8a1687481"} err="failed to get container status \"bcf8bd18b57d1979680d2cee95f2bbc77a3d92a8e900587077fe82c8a1687481\": rpc error: code = NotFound desc = could not find container \"bcf8bd18b57d1979680d2cee95f2bbc77a3d92a8e900587077fe82c8a1687481\": container with ID starting with bcf8bd18b57d1979680d2cee95f2bbc77a3d92a8e900587077fe82c8a1687481 not found: ID does not exist" Nov 02 17:51:24.512201 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.512176 5601 scope.go:122] "RemoveContainer" containerID="0ac9b3d79618ca0a11e8e34ef33bb38a7b796d1d384537e9f7ecfea640fe00e2" Nov 02 17:51:24.512439 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:51:24.512420 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac9b3d79618ca0a11e8e34ef33bb38a7b796d1d384537e9f7ecfea640fe00e2\": container with ID starting with 0ac9b3d79618ca0a11e8e34ef33bb38a7b796d1d384537e9f7ecfea640fe00e2 not found: ID does not exist" containerID="0ac9b3d79618ca0a11e8e34ef33bb38a7b796d1d384537e9f7ecfea640fe00e2" Nov 02 17:51:24.512493 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.512449 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac9b3d79618ca0a11e8e34ef33bb38a7b796d1d384537e9f7ecfea640fe00e2"} err="failed to get container status \"0ac9b3d79618ca0a11e8e34ef33bb38a7b796d1d384537e9f7ecfea640fe00e2\": rpc error: code = NotFound desc = could not find container \"0ac9b3d79618ca0a11e8e34ef33bb38a7b796d1d384537e9f7ecfea640fe00e2\": container with ID starting with 0ac9b3d79618ca0a11e8e34ef33bb38a7b796d1d384537e9f7ecfea640fe00e2 not found: ID does not exist" Nov 02 17:51:24.517894 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.517824 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk"] Nov 02 17:51:24.532574 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:24.532541 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-dd1cba-openshift-default-6788f4f759-5x5vk"] Nov 02 17:51:25.437469 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.437303 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a70cf3-0372-41d0-a78f-c6e396962041" path="/var/lib/kubelet/pods/a5a70cf3-0372-41d0-a78f-c6e396962041/volumes" Nov 02 17:51:25.473415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.473378 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265"] Nov 02 17:51:25.473985 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.473968 5601 cpu_manager.go:414] "containerMap: removing container" podUID="a5a70cf3-0372-41d0-a78f-c6e396962041" containerName="istio-proxy" Nov 02 17:51:25.473985 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.473985 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a70cf3-0372-41d0-a78f-c6e396962041" containerName="istio-proxy" Nov 02 17:51:25.474134 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.473995 5601 cpu_manager.go:414] "containerMap: removing container" podUID="a5a70cf3-0372-41d0-a78f-c6e396962041" containerName="istio-proxy" Nov 02 17:51:25.474134 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.474003 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a70cf3-0372-41d0-a78f-c6e396962041" containerName="istio-proxy" Nov 02 17:51:25.474134 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.474029 5601 cpu_manager.go:414] "containerMap: removing container" podUID="e1158d50-0ccc-4cf3-b7f5-967e25c61bab" containerName="maas-api" Nov 02 17:51:25.474134 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.474034 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1158d50-0ccc-4cf3-b7f5-967e25c61bab" containerName="maas-api" Nov 02 17:51:25.474134 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.474045 5601 cpu_manager.go:414] "containerMap: removing container" podUID="3d301da1-7d1b-4b15-af60-61344075edb7" containerName="maas-api" Nov 02 17:51:25.474134 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.474050 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d301da1-7d1b-4b15-af60-61344075edb7" containerName="maas-api" Nov 02 17:51:25.474429 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.474150 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1158d50-0ccc-4cf3-b7f5-967e25c61bab" containerName="maas-api" Nov 02 17:51:25.474429 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.474158 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d301da1-7d1b-4b15-af60-61344075edb7" containerName="maas-api" Nov 02 17:51:25.474429 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.474167 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f62a981-07b4-4450-9013-959356a33a36" containerName="istio-proxy" Nov 02 17:51:25.474429 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.474174 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a70cf3-0372-41d0-a78f-c6e396962041" containerName="istio-proxy" Nov 02 17:51:25.474429 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.474181 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a70cf3-0372-41d0-a78f-c6e396962041" containerName="istio-proxy" Nov 02 17:51:25.479434 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.479398 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.483739 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.483703 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-token\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.483929 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.483892 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istiod-ca-cert\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.485353 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.484170 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-workload-socket\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.485353 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.484265 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-credential-socket\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.485353 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.484294 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dktl2\" (UniqueName: \"kubernetes.io/projected/3465ffa7-2dbd-41c8-99e6-3e484be099d7-kube-api-access-dktl2\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.485353 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.484324 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-data\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.485353 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.484419 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-podinfo\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.485353 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.484773 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-workload-certs\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.485353 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.484848 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-envoy\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.492080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.490667 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6098d0-openshift-default-dockercfg-9ktsz\"" Nov 02 17:51:25.500126 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.500004 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265"] Nov 02 17:51:25.585878 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.585839 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-workload-socket\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.585878 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.585895 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-credential-socket\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.586158 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.585912 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dktl2\" (UniqueName: \"kubernetes.io/projected/3465ffa7-2dbd-41c8-99e6-3e484be099d7-kube-api-access-dktl2\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.586158 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.585931 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-data\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.586158 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.585963 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-podinfo\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.586158 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.586022 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-workload-certs\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.586158 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.586055 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-envoy\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.586158 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.586076 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-token\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.586158 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.586129 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istiod-ca-cert\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.586503 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.586340 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-workload-socket\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.586973 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.586939 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-workload-certs\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.587092 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.587028 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istiod-ca-cert\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.587242 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.587221 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-data\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.587333 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.587311 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-credential-socket\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.589584 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.589550 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-envoy\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.589857 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.589771 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-podinfo\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.595985 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.595949 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-token\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.596249 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.596228 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dktl2\" (UniqueName: \"kubernetes.io/projected/3465ffa7-2dbd-41c8-99e6-3e484be099d7-kube-api-access-dktl2\") pod \"e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:25.810691 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:25.808894 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:26.156930 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:26.156903 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265"] Nov 02 17:51:26.160551 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:26.160504 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:51:26.160704 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:26.160587 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:51:26.160704 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:26.160644 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:51:26.530904 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:26.530860 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" event={"ID":"3465ffa7-2dbd-41c8-99e6-3e484be099d7","Type":"ContainerStarted","Data":"caa28cabe1402d18625dd9d3dbe64e91775577eed6ef6635c6ed4a914028f554"} Nov 02 17:51:26.531507 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:26.530930 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" event={"ID":"3465ffa7-2dbd-41c8-99e6-3e484be099d7","Type":"ContainerStarted","Data":"58732885cf782bd09e0b26e35ee4e2eaf3dfe7e4e1eba5ad84f99d268cfed1c4"} Nov 02 17:51:26.574006 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:26.573946 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" podStartSLOduration=1.573927888 podStartE2EDuration="1.573927888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:26.569729921 +0000 UTC m=+1051.260783934" watchObservedRunningTime="2026-11-02 17:51:26.573927888 +0000 UTC m=+1051.264981900" Nov 02 17:51:26.812517 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:26.811974 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:27.816564 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:27.816534 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:28.554098 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:28.554056 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:28.555176 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:28.555148 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:51:32.445915 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:32.445879 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs"] Nov 02 17:51:32.450316 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:32.450292 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs" Nov 02 17:51:32.462533 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:32.462481 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs"] Nov 02 17:51:32.464471 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:32.464447 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-7dbdb8-dockercfg-xp2w2\"" Nov 02 17:51:32.464926 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:32.464906 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-7dbdb8\"" Nov 02 17:51:32.526028 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:32.525991 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p29nx\" (UniqueName: \"kubernetes.io/projected/cdb5b8e3-8e9c-463c-b122-1724581fdd8f-kube-api-access-p29nx\") pod \"payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs\" (UID: \"cdb5b8e3-8e9c-463c-b122-1724581fdd8f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs" Nov 02 17:51:32.526247 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:32.526216 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb5b8e3-8e9c-463c-b122-1724581fdd8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs\" (UID: \"cdb5b8e3-8e9c-463c-b122-1724581fdd8f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs" Nov 02 17:51:32.652715 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:32.650172 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p29nx\" (UniqueName: \"kubernetes.io/projected/cdb5b8e3-8e9c-463c-b122-1724581fdd8f-kube-api-access-p29nx\") pod \"payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs\" (UID: \"cdb5b8e3-8e9c-463c-b122-1724581fdd8f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs" Nov 02 17:51:32.652715 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:32.650389 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb5b8e3-8e9c-463c-b122-1724581fdd8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs\" (UID: \"cdb5b8e3-8e9c-463c-b122-1724581fdd8f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs" Nov 02 17:51:32.652715 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:32.651384 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb5b8e3-8e9c-463c-b122-1724581fdd8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs\" (UID: \"cdb5b8e3-8e9c-463c-b122-1724581fdd8f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs" Nov 02 17:51:32.699489 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:32.699290 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p29nx\" (UniqueName: \"kubernetes.io/projected/cdb5b8e3-8e9c-463c-b122-1724581fdd8f-kube-api-access-p29nx\") pod \"payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs\" (UID: \"cdb5b8e3-8e9c-463c-b122-1724581fdd8f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs" Nov 02 17:51:32.770010 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:32.769979 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs" Nov 02 17:51:33.066113 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:33.066019 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs"] Nov 02 17:51:33.078653 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:51:33.078554 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb5b8e3_8e9c_463c_b122_1724581fdd8f.slice/crio-dfd82dd07f05c5621d6f98aeeca013b4be1bb1b68837d0702da6acf3468e5845 WatchSource:0}: Error finding container dfd82dd07f05c5621d6f98aeeca013b4be1bb1b68837d0702da6acf3468e5845: Status 404 returned error can't find the container with id dfd82dd07f05c5621d6f98aeeca013b4be1bb1b68837d0702da6acf3468e5845 Nov 02 17:51:33.651182 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:33.651151 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs" event={"ID":"cdb5b8e3-8e9c-463c-b122-1724581fdd8f","Type":"ContainerStarted","Data":"20075eadfd26461bd6f4c536a0149b40eca237c541706364b8c817a89b22f43b"} Nov 02 17:51:33.651571 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:33.651199 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs" event={"ID":"cdb5b8e3-8e9c-463c-b122-1724581fdd8f","Type":"ContainerStarted","Data":"dfd82dd07f05c5621d6f98aeeca013b4be1bb1b68837d0702da6acf3468e5845"} Nov 02 17:51:33.651571 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:33.651238 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs" Nov 02 17:51:33.670736 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:33.670683 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs" podStartSLOduration=1.670669121 podStartE2EDuration="1.670669121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:33.669835839 +0000 UTC m=+1058.360889852" watchObservedRunningTime="2026-11-02 17:51:33.670669121 +0000 UTC m=+1058.361723131" Nov 02 17:51:34.485500 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:51:34.485464 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:51:35.404349 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:35.404309 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt"] Nov 02 17:51:35.409440 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:35.409413 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" Nov 02 17:51:35.428581 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:35.428363 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-e4fdc1-serving-cert\"" Nov 02 17:51:35.433878 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:35.433852 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt"] Nov 02 17:51:35.494075 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:35.494038 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbfr6\" (UniqueName: \"kubernetes.io/projected/466397d6-4f0f-4b1c-98d3-93903456336a-kube-api-access-wbfr6\") pod \"maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt\" (UID: \"466397d6-4f0f-4b1c-98d3-93903456336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" Nov 02 17:51:35.494283 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:35.494187 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466397d6-4f0f-4b1c-98d3-93903456336a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt\" (UID: \"466397d6-4f0f-4b1c-98d3-93903456336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" Nov 02 17:51:35.494333 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:35.494318 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/466397d6-4f0f-4b1c-98d3-93903456336a-metrics-certs\") pod \"maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt\" (UID: \"466397d6-4f0f-4b1c-98d3-93903456336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" Nov 02 17:51:35.494393 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:35.494381 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466397d6-4f0f-4b1c-98d3-93903456336a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt\" (UID: \"466397d6-4f0f-4b1c-98d3-93903456336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" Nov 02 17:51:35.605681 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:35.596663 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466397d6-4f0f-4b1c-98d3-93903456336a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt\" (UID: \"466397d6-4f0f-4b1c-98d3-93903456336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" Nov 02 17:51:35.605681 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:35.596795 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/466397d6-4f0f-4b1c-98d3-93903456336a-metrics-certs\") pod \"maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt\" (UID: \"466397d6-4f0f-4b1c-98d3-93903456336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" Nov 02 17:51:35.605681 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:35.596926 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466397d6-4f0f-4b1c-98d3-93903456336a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt\" (UID: \"466397d6-4f0f-4b1c-98d3-93903456336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" Nov 02 17:51:35.605681 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:35.602393 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466397d6-4f0f-4b1c-98d3-93903456336a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt\" (UID: \"466397d6-4f0f-4b1c-98d3-93903456336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" Nov 02 17:51:35.607263 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:35.606248 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbfr6\" (UniqueName: \"kubernetes.io/projected/466397d6-4f0f-4b1c-98d3-93903456336a-kube-api-access-wbfr6\") pod \"maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt\" (UID: \"466397d6-4f0f-4b1c-98d3-93903456336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" Nov 02 17:51:35.611471 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:35.611440 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/466397d6-4f0f-4b1c-98d3-93903456336a-metrics-certs\") pod \"maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt\" (UID: \"466397d6-4f0f-4b1c-98d3-93903456336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" Nov 02 17:51:35.613820 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:35.613787 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466397d6-4f0f-4b1c-98d3-93903456336a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt\" (UID: \"466397d6-4f0f-4b1c-98d3-93903456336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" Nov 02 17:51:35.619045 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:35.619016 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbfr6\" (UniqueName: \"kubernetes.io/projected/466397d6-4f0f-4b1c-98d3-93903456336a-kube-api-access-wbfr6\") pod \"maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt\" (UID: \"466397d6-4f0f-4b1c-98d3-93903456336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" Nov 02 17:51:35.729606 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:35.729572 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" Nov 02 17:51:36.032408 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:36.017443 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt"] Nov 02 17:51:36.679330 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:36.679291 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" event={"ID":"466397d6-4f0f-4b1c-98d3-93903456336a","Type":"ContainerStarted","Data":"76dda50a601e35ce19d89bd0f2f088a6cefacc223206c307d43ff6d6c868ec8d"} Nov 02 17:51:36.679748 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:36.679353 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" event={"ID":"466397d6-4f0f-4b1c-98d3-93903456336a","Type":"ContainerStarted","Data":"1b6cfc2d5ab95908677f0ad44a95a640ba565cfda5abeb1821a6e5821709acaf"} Nov 02 17:51:36.679748 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:36.679382 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" Nov 02 17:51:36.702768 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:36.702715 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" podStartSLOduration=1.281288331 podStartE2EDuration="1.702701017s" totalImagesPullingTime="421.412686ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:36.700041818 +0000 UTC m=+1061.391095855" watchObservedRunningTime="2026-11-02 17:51:36.702701017 +0000 UTC m=+1061.393755028" Nov 02 17:51:39.031610 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:39.031579 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq"] Nov 02 17:51:39.041913 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:39.041889 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" Nov 02 17:51:39.047120 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:39.047092 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-6098d0-serving-cert\"" Nov 02 17:51:39.052057 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:39.052031 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq"] Nov 02 17:51:39.168092 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:39.168058 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffeee2eb-8d38-40dd-b000-4863eed2bd86-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq\" (UID: \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" Nov 02 17:51:39.168266 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:39.168227 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxwwm\" (UniqueName: \"kubernetes.io/projected/ffeee2eb-8d38-40dd-b000-4863eed2bd86-kube-api-access-sxwwm\") pod \"maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq\" (UID: \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" Nov 02 17:51:39.168340 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:39.168300 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffeee2eb-8d38-40dd-b000-4863eed2bd86-metrics-certs\") pod \"maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq\" (UID: \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" Nov 02 17:51:39.168384 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:39.168365 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffeee2eb-8d38-40dd-b000-4863eed2bd86-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq\" (UID: \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" Nov 02 17:51:39.269607 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:39.269571 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffeee2eb-8d38-40dd-b000-4863eed2bd86-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq\" (UID: \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" Nov 02 17:51:39.269848 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:39.269729 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwwm\" (UniqueName: \"kubernetes.io/projected/ffeee2eb-8d38-40dd-b000-4863eed2bd86-kube-api-access-sxwwm\") pod \"maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq\" (UID: \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" Nov 02 17:51:39.269848 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:39.269802 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffeee2eb-8d38-40dd-b000-4863eed2bd86-metrics-certs\") pod \"maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq\" (UID: \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" Nov 02 17:51:39.269959 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:39.269856 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffeee2eb-8d38-40dd-b000-4863eed2bd86-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq\" (UID: \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" Nov 02 17:51:39.270134 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:51:39.270110 5601 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-serving-cert: secret "maas-api-e2e-ipp-a-6098d0-serving-cert" not found Nov 02 17:51:39.270245 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:51:39.270180 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffeee2eb-8d38-40dd-b000-4863eed2bd86-maas-api-tls podName:ffeee2eb-8d38-40dd-b000-4863eed2bd86 nodeName:}" failed. No retries permitted until 2026-11-02 17:51:39.770160661 +0000 UTC m=+1064.461214667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ffeee2eb-8d38-40dd-b000-4863eed2bd86-maas-api-tls") pod "maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" (UID: "ffeee2eb-8d38-40dd-b000-4863eed2bd86") : secret "maas-api-e2e-ipp-a-6098d0-serving-cert" not found Nov 02 17:51:39.270424 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:39.270400 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffeee2eb-8d38-40dd-b000-4863eed2bd86-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq\" (UID: \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" Nov 02 17:51:39.273181 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:39.273154 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffeee2eb-8d38-40dd-b000-4863eed2bd86-metrics-certs\") pod \"maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq\" (UID: \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" Nov 02 17:51:39.281164 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:39.281140 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxwwm\" (UniqueName: \"kubernetes.io/projected/ffeee2eb-8d38-40dd-b000-4863eed2bd86-kube-api-access-sxwwm\") pod \"maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq\" (UID: \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" Nov 02 17:51:39.783725 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:39.783684 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffeee2eb-8d38-40dd-b000-4863eed2bd86-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq\" (UID: \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" Nov 02 17:51:39.786412 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:39.786384 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffeee2eb-8d38-40dd-b000-4863eed2bd86-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq\" (UID: \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" Nov 02 17:51:39.960256 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:39.960222 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" Nov 02 17:51:40.289010 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:40.288976 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq"] Nov 02 17:51:40.289474 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:51:40.289447 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffeee2eb_8d38_40dd_b000_4863eed2bd86.slice/crio-4465311a8f761d50ae727a97407e4926ae824725f1adbb067e58ddfde13b3756 WatchSource:0}: Error finding container 4465311a8f761d50ae727a97407e4926ae824725f1adbb067e58ddfde13b3756: Status 404 returned error can't find the container with id 4465311a8f761d50ae727a97407e4926ae824725f1adbb067e58ddfde13b3756 Nov 02 17:51:40.743208 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:40.742995 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" event={"ID":"ffeee2eb-8d38-40dd-b000-4863eed2bd86","Type":"ContainerStarted","Data":"4465311a8f761d50ae727a97407e4926ae824725f1adbb067e58ddfde13b3756"} Nov 02 17:51:41.765639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:41.759678 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" event={"ID":"ffeee2eb-8d38-40dd-b000-4863eed2bd86","Type":"ContainerStarted","Data":"abcfc78a15c15b93ae709a0a158d842d21636e8d3ccbd1004238fe1ded18d7bf"} Nov 02 17:51:41.765639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:41.760863 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" Nov 02 17:51:41.793481 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:41.793413 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" podStartSLOduration=2.386653044 podStartE2EDuration="2.793393957s" totalImagesPullingTime="406.740913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:41.787773592 +0000 UTC m=+1066.478827602" watchObservedRunningTime="2026-11-02 17:51:41.793393957 +0000 UTC m=+1066.484447970" Nov 02 17:51:42.692288 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:42.692257 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" Nov 02 17:51:44.663089 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:44.663061 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs" Nov 02 17:51:48.785758 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:48.785727 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" Nov 02 17:51:49.339465 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:49.339421 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt"] Nov 02 17:51:49.339747 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:49.339714 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" podUID="54a717b2-8169-498d-90e3-d8499f9af4de" containerName="maas-api" containerID="cri-o://b120b783d47864720ae7d90c8889586cd342b21aca969d29967019d54baf2127" gracePeriod=30 Nov 02 17:51:49.813952 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:49.813924 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" Nov 02 17:51:49.844692 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:49.844659 5601 generic.go:356] "Generic (PLEG): container finished" podID="54a717b2-8169-498d-90e3-d8499f9af4de" containerID="b120b783d47864720ae7d90c8889586cd342b21aca969d29967019d54baf2127" exitCode=0 Nov 02 17:51:49.844872 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:49.844742 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" Nov 02 17:51:49.844872 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:49.844759 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" event={"ID":"54a717b2-8169-498d-90e3-d8499f9af4de","Type":"ContainerDied","Data":"b120b783d47864720ae7d90c8889586cd342b21aca969d29967019d54baf2127"} Nov 02 17:51:49.844872 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:49.844803 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt" event={"ID":"54a717b2-8169-498d-90e3-d8499f9af4de","Type":"ContainerDied","Data":"c48127e5e0b326c4cf399b12d885a75bebbda413894aa5246b91e0d01a39a173"} Nov 02 17:51:49.844872 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:49.844827 5601 scope.go:122] "RemoveContainer" containerID="b120b783d47864720ae7d90c8889586cd342b21aca969d29967019d54baf2127" Nov 02 17:51:49.861862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:49.861769 5601 scope.go:122] "RemoveContainer" containerID="b120b783d47864720ae7d90c8889586cd342b21aca969d29967019d54baf2127" Nov 02 17:51:49.863119 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:51:49.863082 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b120b783d47864720ae7d90c8889586cd342b21aca969d29967019d54baf2127\": container with ID starting with b120b783d47864720ae7d90c8889586cd342b21aca969d29967019d54baf2127 not found: ID does not exist" containerID="b120b783d47864720ae7d90c8889586cd342b21aca969d29967019d54baf2127" Nov 02 17:51:49.863278 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:49.863135 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b120b783d47864720ae7d90c8889586cd342b21aca969d29967019d54baf2127"} err="failed to get container status \"b120b783d47864720ae7d90c8889586cd342b21aca969d29967019d54baf2127\": rpc error: code = NotFound desc = could not find container \"b120b783d47864720ae7d90c8889586cd342b21aca969d29967019d54baf2127\": container with ID starting with b120b783d47864720ae7d90c8889586cd342b21aca969d29967019d54baf2127 not found: ID does not exist" Nov 02 17:51:49.901022 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:49.900984 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a717b2-8169-498d-90e3-d8499f9af4de-kube-api-access-4gcwc\" (UniqueName: \"kubernetes.io/projected/54a717b2-8169-498d-90e3-d8499f9af4de-kube-api-access-4gcwc\") pod \"54a717b2-8169-498d-90e3-d8499f9af4de\" (UID: \"54a717b2-8169-498d-90e3-d8499f9af4de\") " Nov 02 17:51:49.901219 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:49.901031 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54a717b2-8169-498d-90e3-d8499f9af4de-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54a717b2-8169-498d-90e3-d8499f9af4de-openshift-service-ca\") pod \"54a717b2-8169-498d-90e3-d8499f9af4de\" (UID: \"54a717b2-8169-498d-90e3-d8499f9af4de\") " Nov 02 17:51:49.901219 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:49.901149 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54a717b2-8169-498d-90e3-d8499f9af4de-metrics-certs\" (UniqueName: \"kubernetes.io/secret/54a717b2-8169-498d-90e3-d8499f9af4de-metrics-certs\") pod \"54a717b2-8169-498d-90e3-d8499f9af4de\" (UID: \"54a717b2-8169-498d-90e3-d8499f9af4de\") " Nov 02 17:51:49.901350 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:49.901278 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54a717b2-8169-498d-90e3-d8499f9af4de-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54a717b2-8169-498d-90e3-d8499f9af4de-maas-api-tls\") pod \"54a717b2-8169-498d-90e3-d8499f9af4de\" (UID: \"54a717b2-8169-498d-90e3-d8499f9af4de\") " Nov 02 17:51:49.901485 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:49.901454 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a717b2-8169-498d-90e3-d8499f9af4de-openshift-service-ca" pod "54a717b2-8169-498d-90e3-d8499f9af4de" (UID: "54a717b2-8169-498d-90e3-d8499f9af4de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:49.901837 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:49.901783 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54a717b2-8169-498d-90e3-d8499f9af4de-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:49.903228 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:49.903201 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a717b2-8169-498d-90e3-d8499f9af4de-metrics-certs" pod "54a717b2-8169-498d-90e3-d8499f9af4de" (UID: "54a717b2-8169-498d-90e3-d8499f9af4de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:49.903322 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:49.903223 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a717b2-8169-498d-90e3-d8499f9af4de-kube-api-access-4gcwc" pod "54a717b2-8169-498d-90e3-d8499f9af4de" (UID: "54a717b2-8169-498d-90e3-d8499f9af4de"). InnerVolumeSpecName "kube-api-access-4gcwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:49.903322 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:49.903264 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a717b2-8169-498d-90e3-d8499f9af4de-maas-api-tls" pod "54a717b2-8169-498d-90e3-d8499f9af4de" (UID: "54a717b2-8169-498d-90e3-d8499f9af4de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:50.003449 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:50.003411 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gcwc\" (UniqueName: \"kubernetes.io/projected/54a717b2-8169-498d-90e3-d8499f9af4de-kube-api-access-4gcwc\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:50.003449 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:50.003446 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54a717b2-8169-498d-90e3-d8499f9af4de-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:50.003449 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:50.003456 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54a717b2-8169-498d-90e3-d8499f9af4de-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:50.179946 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:50.179881 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt"] Nov 02 17:51:50.184785 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:50.184749 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd2738-6b9ccddf96-nnqgt"] Nov 02 17:51:51.429147 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:51.429107 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54a717b2-8169-498d-90e3-d8499f9af4de" path="/var/lib/kubelet/pods/54a717b2-8169-498d-90e3-d8499f9af4de/volumes" Nov 02 17:51:55.924303 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:55.924267 5601 scope.go:122] "RemoveContainer" containerID="3850531e248af63bbe4753a7e34aa8a09598ceb62ed2979fda57ee2d080e96ab" Nov 02 17:51:58.064401 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.064369 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84"] Nov 02 17:51:58.065105 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.065085 5601 cpu_manager.go:414] "containerMap: removing container" podUID="54a717b2-8169-498d-90e3-d8499f9af4de" containerName="maas-api" Nov 02 17:51:58.065105 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.065105 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a717b2-8169-498d-90e3-d8499f9af4de" containerName="maas-api" Nov 02 17:51:58.065295 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.065232 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a717b2-8169-498d-90e3-d8499f9af4de" containerName="maas-api" Nov 02 17:51:58.070306 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.070286 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.082368 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.082334 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-4a5f09-openshift-default-dockercfg-hmfp5\"" Nov 02 17:51:58.116683 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.116651 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84"] Nov 02 17:51:58.210641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.196996 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-envoy\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.210641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.197173 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istiod-ca-cert\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.210641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.197212 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-credential-socket\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.210641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.197267 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-token\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.210641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.197303 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-workload-certs\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.210641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.197412 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-data\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.210641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.197519 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-podinfo\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.210641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.197598 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-workload-socket\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.210641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.197660 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xx2n\" (UniqueName: \"kubernetes.io/projected/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-kube-api-access-2xx2n\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.322850 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.322753 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-podinfo\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.325858 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.323114 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-workload-socket\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.326181 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.326154 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xx2n\" (UniqueName: \"kubernetes.io/projected/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-kube-api-access-2xx2n\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.326311 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.326296 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-envoy\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.326541 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.326526 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istiod-ca-cert\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.326670 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.326656 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-credential-socket\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.326811 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.326796 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-token\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.326924 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.326910 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-workload-certs\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.327106 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.327093 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-data\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.327787 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.327765 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-data\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.342469 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.342440 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-workload-socket\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.383822 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.347948 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-workload-certs\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.383822 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.348185 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-credential-socket\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.384062 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.352931 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-envoy\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.384062 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.365321 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xx2n\" (UniqueName: \"kubernetes.io/projected/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-kube-api-access-2xx2n\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.384062 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.383736 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istiod-ca-cert\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.387041 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.387014 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-token\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.391800 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.391772 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-podinfo\") pod \"e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:58.689570 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:58.689535 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:51:59.028962 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:59.028933 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84"] Nov 02 17:51:59.031557 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:51:59.029420 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d7e67db_b4ab_496e_b7f0_d1af7f4c7b40.slice/crio-8c673a43ad3ad02a235185bb113a86c2b8c73d8ce490a39c228fbca3411a120b WatchSource:0}: Error finding container 8c673a43ad3ad02a235185bb113a86c2b8c73d8ce490a39c228fbca3411a120b: Status 404 returned error can't find the container with id 8c673a43ad3ad02a235185bb113a86c2b8c73d8ce490a39c228fbca3411a120b Nov 02 17:51:59.032473 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:59.032428 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:51:59.032581 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:59.032538 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:51:59.032702 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:59.032581 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:51:59.940785 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:59.940734 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" event={"ID":"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40","Type":"ContainerStarted","Data":"f207fee6549bcc23658d0a936a3137680497dc455e647eee3b28470c1ba95e9d"} Nov 02 17:51:59.941180 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:59.940802 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" event={"ID":"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40","Type":"ContainerStarted","Data":"8c673a43ad3ad02a235185bb113a86c2b8c73d8ce490a39c228fbca3411a120b"} Nov 02 17:51:59.974137 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:51:59.974085 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" podStartSLOduration=1.9740725399999999 podStartE2EDuration="1.97407254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:59.970524798 +0000 UTC m=+1084.661578811" watchObservedRunningTime="2026-11-02 17:51:59.97407254 +0000 UTC m=+1084.665126555" Nov 02 17:52:00.690083 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:00.690050 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:52:00.695661 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:00.695633 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:52:00.947908 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:00.947824 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:52:00.948899 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:00.948880 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:52:04.487066 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:52:04.487027 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:52:07.837701 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:07.833906 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft"] Nov 02 17:52:07.837701 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:07.834169 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" podUID="5c1fcd8f-3977-4d4b-826c-38150e08228c" containerName="istio-proxy" containerID="cri-o://4dc8ac7a3054619e9036846edbf077daab0d8aa87061a7552e1ba57ff52abeb1" gracePeriod=30 Nov 02 17:52:07.837701 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:07.837452 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz"] Nov 02 17:52:07.838286 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:07.837758 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" podUID="5eb03e7c-1886-4f63-ab93-312dbe0913cf" containerName="istio-proxy" containerID="cri-o://7d0018b44e7ff07a2071ebaf1f8c53d85d36a6baf02a4be71cfbb9430dc80ecb" gracePeriod=30 Nov 02 17:52:13.188682 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.188634 5601 generic.go:356] "Generic (PLEG): container finished" podID="5c1fcd8f-3977-4d4b-826c-38150e08228c" containerID="4dc8ac7a3054619e9036846edbf077daab0d8aa87061a7552e1ba57ff52abeb1" exitCode=0 Nov 02 17:52:13.189181 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.188919 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" event={"ID":"5c1fcd8f-3977-4d4b-826c-38150e08228c","Type":"ContainerDied","Data":"4dc8ac7a3054619e9036846edbf077daab0d8aa87061a7552e1ba57ff52abeb1"} Nov 02 17:52:13.189181 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.189021 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" event={"ID":"5c1fcd8f-3977-4d4b-826c-38150e08228c","Type":"ContainerDied","Data":"18b78f655ff3f4273b9927441b8dd81567c1983785a637b4bb039c419fa6559a"} Nov 02 17:52:13.189181 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.189053 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18b78f655ff3f4273b9927441b8dd81567c1983785a637b4bb039c419fa6559a" Nov 02 17:52:13.192461 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.192435 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:52:13.192695 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.192669 5601 generic.go:356] "Generic (PLEG): container finished" podID="5eb03e7c-1886-4f63-ab93-312dbe0913cf" containerID="7d0018b44e7ff07a2071ebaf1f8c53d85d36a6baf02a4be71cfbb9430dc80ecb" exitCode=0 Nov 02 17:52:13.192800 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.192736 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" event={"ID":"5eb03e7c-1886-4f63-ab93-312dbe0913cf","Type":"ContainerDied","Data":"7d0018b44e7ff07a2071ebaf1f8c53d85d36a6baf02a4be71cfbb9430dc80ecb"} Nov 02 17:52:13.282540 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.282509 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:52:13.302543 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.302506 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-token\" (UniqueName: \"kubernetes.io/projected/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-token\") pod \"5c1fcd8f-3977-4d4b-826c-38150e08228c\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " Nov 02 17:52:13.302758 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.302549 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-podinfo\") pod \"5c1fcd8f-3977-4d4b-826c-38150e08228c\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " Nov 02 17:52:13.302758 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.302580 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-workload-socket\") pod \"5c1fcd8f-3977-4d4b-826c-38150e08228c\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " Nov 02 17:52:13.302758 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.302744 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-envoy\") pod \"5c1fcd8f-3977-4d4b-826c-38150e08228c\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " Nov 02 17:52:13.302936 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.302790 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-data\") pod \"5c1fcd8f-3977-4d4b-826c-38150e08228c\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " Nov 02 17:52:13.302936 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.302869 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-workload-certs\") pod \"5c1fcd8f-3977-4d4b-826c-38150e08228c\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " Nov 02 17:52:13.302936 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.302913 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c1fcd8f-3977-4d4b-826c-38150e08228c-kube-api-access-mgfrf\" (UniqueName: \"kubernetes.io/projected/5c1fcd8f-3977-4d4b-826c-38150e08228c-kube-api-access-mgfrf\") pod \"5c1fcd8f-3977-4d4b-826c-38150e08228c\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " Nov 02 17:52:13.303086 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.302975 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c1fcd8f-3977-4d4b-826c-38150e08228c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1fcd8f-3977-4d4b-826c-38150e08228c-istiod-ca-cert\") pod \"5c1fcd8f-3977-4d4b-826c-38150e08228c\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " Nov 02 17:52:13.303086 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.303039 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-credential-socket\") pod \"5c1fcd8f-3977-4d4b-826c-38150e08228c\" (UID: \"5c1fcd8f-3977-4d4b-826c-38150e08228c\") " Nov 02 17:52:13.303202 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.303168 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-data" pod "5c1fcd8f-3977-4d4b-826c-38150e08228c" (UID: "5c1fcd8f-3977-4d4b-826c-38150e08228c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:13.303264 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.303179 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-workload-socket" pod "5c1fcd8f-3977-4d4b-826c-38150e08228c" (UID: "5c1fcd8f-3977-4d4b-826c-38150e08228c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:13.303435 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.303413 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-credential-socket" pod "5c1fcd8f-3977-4d4b-826c-38150e08228c" (UID: "5c1fcd8f-3977-4d4b-826c-38150e08228c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:13.303660 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.303578 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-workload-certs" pod "5c1fcd8f-3977-4d4b-826c-38150e08228c" (UID: "5c1fcd8f-3977-4d4b-826c-38150e08228c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:13.305987 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.303949 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c1fcd8f-3977-4d4b-826c-38150e08228c-istiod-ca-cert" pod "5c1fcd8f-3977-4d4b-826c-38150e08228c" (UID: "5c1fcd8f-3977-4d4b-826c-38150e08228c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:13.310851 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.306570 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-envoy" pod "5c1fcd8f-3977-4d4b-826c-38150e08228c" (UID: "5c1fcd8f-3977-4d4b-826c-38150e08228c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:13.310851 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.306705 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1fcd8f-3977-4d4b-826c-38150e08228c-kube-api-access-mgfrf" pod "5c1fcd8f-3977-4d4b-826c-38150e08228c" (UID: "5c1fcd8f-3977-4d4b-826c-38150e08228c"). InnerVolumeSpecName "kube-api-access-mgfrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:13.310851 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.307919 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:13.310851 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.307960 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:13.310851 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.307977 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:13.310851 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.308036 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:13.310851 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.308053 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgfrf\" (UniqueName: \"kubernetes.io/projected/5c1fcd8f-3977-4d4b-826c-38150e08228c-kube-api-access-mgfrf\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:13.310851 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.308070 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1fcd8f-3977-4d4b-826c-38150e08228c-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:13.310851 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.308097 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1fcd8f-3977-4d4b-826c-38150e08228c-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:13.310851 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.308587 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-token" pod "5c1fcd8f-3977-4d4b-826c-38150e08228c" (UID: "5c1fcd8f-3977-4d4b-826c-38150e08228c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:13.310851 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.308898 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-podinfo" pod "5c1fcd8f-3977-4d4b-826c-38150e08228c" (UID: "5c1fcd8f-3977-4d4b-826c-38150e08228c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:52:13.409449 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.409416 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-token\" (UniqueName: \"kubernetes.io/projected/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-token\") pod \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " Nov 02 17:52:13.409449 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.409455 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istiod-ca-cert\") pod \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " Nov 02 17:52:13.409704 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.409509 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-workload-certs\") pod \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " Nov 02 17:52:13.409704 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.409641 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-credential-socket\") pod \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " Nov 02 17:52:13.409704 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.409684 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-data\") pod \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " Nov 02 17:52:13.409876 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.409854 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb03e7c-1886-4f63-ab93-312dbe0913cf-kube-api-access-p6zkl\" (UniqueName: \"kubernetes.io/projected/5eb03e7c-1886-4f63-ab93-312dbe0913cf-kube-api-access-p6zkl\") pod \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " Nov 02 17:52:13.409947 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.409868 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-credential-socket" pod "5eb03e7c-1886-4f63-ab93-312dbe0913cf" (UID: "5eb03e7c-1886-4f63-ab93-312dbe0913cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:13.409947 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.409883 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-workload-certs" pod "5eb03e7c-1886-4f63-ab93-312dbe0913cf" (UID: "5eb03e7c-1886-4f63-ab93-312dbe0913cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:13.410055 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.409964 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-data" pod "5eb03e7c-1886-4f63-ab93-312dbe0913cf" (UID: "5eb03e7c-1886-4f63-ab93-312dbe0913cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:13.410055 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.409983 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-podinfo\") pod \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " Nov 02 17:52:13.410055 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.409979 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istiod-ca-cert" pod "5eb03e7c-1886-4f63-ab93-312dbe0913cf" (UID: "5eb03e7c-1886-4f63-ab93-312dbe0913cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:13.410055 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.410014 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-envoy\") pod \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " Nov 02 17:52:13.410309 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.410069 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-workload-socket\") pod \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\" (UID: \"5eb03e7c-1886-4f63-ab93-312dbe0913cf\") " Nov 02 17:52:13.410365 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.410339 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-workload-socket" pod "5eb03e7c-1886-4f63-ab93-312dbe0913cf" (UID: "5eb03e7c-1886-4f63-ab93-312dbe0913cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:13.410792 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.410763 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:13.410935 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.410794 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:13.410935 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.410809 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:13.410935 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.410840 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:13.410935 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.410857 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:13.410935 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.410872 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:13.410935 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.410886 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1fcd8f-3977-4d4b-826c-38150e08228c-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:13.412052 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.412025 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-token" pod "5eb03e7c-1886-4f63-ab93-312dbe0913cf" (UID: "5eb03e7c-1886-4f63-ab93-312dbe0913cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:13.412156 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.412066 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-podinfo" pod "5eb03e7c-1886-4f63-ab93-312dbe0913cf" (UID: "5eb03e7c-1886-4f63-ab93-312dbe0913cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:52:13.412201 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.412167 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb03e7c-1886-4f63-ab93-312dbe0913cf-kube-api-access-p6zkl" pod "5eb03e7c-1886-4f63-ab93-312dbe0913cf" (UID: "5eb03e7c-1886-4f63-ab93-312dbe0913cf"). InnerVolumeSpecName "kube-api-access-p6zkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:13.412201 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.412185 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-envoy" pod "5eb03e7c-1886-4f63-ab93-312dbe0913cf" (UID: "5eb03e7c-1886-4f63-ab93-312dbe0913cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:13.512012 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.511977 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6zkl\" (UniqueName: \"kubernetes.io/projected/5eb03e7c-1886-4f63-ab93-312dbe0913cf-kube-api-access-p6zkl\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:13.512012 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.512010 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:13.512012 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.512022 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:13.512296 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:13.512031 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb03e7c-1886-4f63-ab93-312dbe0913cf-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:14.203520 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:14.203493 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" Nov 02 17:52:14.203520 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:14.203519 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft" Nov 02 17:52:14.204039 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:14.203519 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz" event={"ID":"5eb03e7c-1886-4f63-ab93-312dbe0913cf","Type":"ContainerDied","Data":"5d2225ba69c2aa1b432a1110a183da46346e00981c23c39b05ab58b9a95265f0"} Nov 02 17:52:14.204039 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:14.203580 5601 scope.go:122] "RemoveContainer" containerID="7d0018b44e7ff07a2071ebaf1f8c53d85d36a6baf02a4be71cfbb9430dc80ecb" Nov 02 17:52:14.227774 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:14.226048 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft"] Nov 02 17:52:14.230197 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:14.230171 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-fxnft"] Nov 02 17:52:14.271765 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:14.271727 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz"] Nov 02 17:52:14.274395 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:14.274360 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-7pdvz"] Nov 02 17:52:15.440592 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:15.440555 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c1fcd8f-3977-4d4b-826c-38150e08228c" path="/var/lib/kubelet/pods/5c1fcd8f-3977-4d4b-826c-38150e08228c/volumes" Nov 02 17:52:15.443735 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:15.443704 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb03e7c-1886-4f63-ab93-312dbe0913cf" path="/var/lib/kubelet/pods/5eb03e7c-1886-4f63-ab93-312dbe0913cf/volumes" Nov 02 17:52:20.248929 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:20.248897 5601 generic.go:356] "Generic (PLEG): container finished" podID="3465ffa7-2dbd-41c8-99e6-3e484be099d7" containerID="caa28cabe1402d18625dd9d3dbe64e91775577eed6ef6635c6ed4a914028f554" exitCode=0 Nov 02 17:52:20.249363 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:20.248978 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" event={"ID":"3465ffa7-2dbd-41c8-99e6-3e484be099d7","Type":"ContainerDied","Data":"caa28cabe1402d18625dd9d3dbe64e91775577eed6ef6635c6ed4a914028f554"} Nov 02 17:52:20.249444 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:20.249427 5601 scope.go:122] "RemoveContainer" containerID="caa28cabe1402d18625dd9d3dbe64e91775577eed6ef6635c6ed4a914028f554" Nov 02 17:52:20.251051 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:20.251022 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:52:20.251104 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:20.251096 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:52:20.251138 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:20.251128 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:52:21.273800 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:21.273759 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" event={"ID":"3465ffa7-2dbd-41c8-99e6-3e484be099d7","Type":"ContainerStarted","Data":"4dffe2ea87bd1dcb9c8b148a1de4d31f84dc783a884db3aba19c7edaea1ef949"} Nov 02 17:52:21.817683 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:21.817002 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:52:21.817683 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:21.817131 5601 patch_prober.go:29] interesting pod/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:52:21.817683 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:21.817173 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" podUID="3465ffa7-2dbd-41c8-99e6-3e484be099d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 17:52:22.812088 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:22.812052 5601 patch_prober.go:29] interesting pod/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:52:22.812505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:22.812110 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" podUID="3465ffa7-2dbd-41c8-99e6-3e484be099d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 17:52:23.814672 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:23.814063 5601 patch_prober.go:29] interesting pod/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:52:23.814672 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:23.814131 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" podUID="3465ffa7-2dbd-41c8-99e6-3e484be099d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 17:52:24.812119 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:24.812086 5601 patch_prober.go:29] interesting pod/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:52:24.812310 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:24.812163 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" podUID="3465ffa7-2dbd-41c8-99e6-3e484be099d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 17:52:25.812041 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:25.812003 5601 patch_prober.go:29] interesting pod/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:52:25.812432 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:25.812058 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" podUID="3465ffa7-2dbd-41c8-99e6-3e484be099d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 17:52:26.922246 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:26.922214 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:52:26.922785 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:26.922653 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:52:26.923951 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:26.923930 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:52:33.901275 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:33.901239 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd"] Nov 02 17:52:33.901836 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:33.901822 5601 cpu_manager.go:414] "containerMap: removing container" podUID="5eb03e7c-1886-4f63-ab93-312dbe0913cf" containerName="istio-proxy" Nov 02 17:52:33.901892 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:33.901837 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb03e7c-1886-4f63-ab93-312dbe0913cf" containerName="istio-proxy" Nov 02 17:52:33.901892 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:33.901865 5601 cpu_manager.go:414] "containerMap: removing container" podUID="5c1fcd8f-3977-4d4b-826c-38150e08228c" containerName="istio-proxy" Nov 02 17:52:33.901892 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:33.901870 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c1fcd8f-3977-4d4b-826c-38150e08228c" containerName="istio-proxy" Nov 02 17:52:33.901994 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:33.901953 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb03e7c-1886-4f63-ab93-312dbe0913cf" containerName="istio-proxy" Nov 02 17:52:33.901994 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:33.901968 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c1fcd8f-3977-4d4b-826c-38150e08228c" containerName="istio-proxy" Nov 02 17:52:33.904437 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:33.904419 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" Nov 02 17:52:33.913208 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:33.913181 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d8e217\"" Nov 02 17:52:33.913370 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:33.913220 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d8e217-dockercfg-ctkkn\"" Nov 02 17:52:33.916266 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:33.916241 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd"] Nov 02 17:52:34.080639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:34.077886 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r24n2\" (UniqueName: \"kubernetes.io/projected/5e5069cc-7c3d-4f35-8a37-c874e0a834f3-kube-api-access-r24n2\") pod \"payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd\" (UID: \"5e5069cc-7c3d-4f35-8a37-c874e0a834f3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" Nov 02 17:52:34.080639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:34.077976 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5069cc-7c3d-4f35-8a37-c874e0a834f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd\" (UID: \"5e5069cc-7c3d-4f35-8a37-c874e0a834f3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" Nov 02 17:52:34.182857 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:34.179712 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r24n2\" (UniqueName: \"kubernetes.io/projected/5e5069cc-7c3d-4f35-8a37-c874e0a834f3-kube-api-access-r24n2\") pod \"payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd\" (UID: \"5e5069cc-7c3d-4f35-8a37-c874e0a834f3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" Nov 02 17:52:34.182857 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:34.179804 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5069cc-7c3d-4f35-8a37-c874e0a834f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd\" (UID: \"5e5069cc-7c3d-4f35-8a37-c874e0a834f3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" Nov 02 17:52:34.182857 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:34.180975 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5069cc-7c3d-4f35-8a37-c874e0a834f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd\" (UID: \"5e5069cc-7c3d-4f35-8a37-c874e0a834f3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" Nov 02 17:52:34.190533 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:34.190507 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r24n2\" (UniqueName: \"kubernetes.io/projected/5e5069cc-7c3d-4f35-8a37-c874e0a834f3-kube-api-access-r24n2\") pod \"payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd\" (UID: \"5e5069cc-7c3d-4f35-8a37-c874e0a834f3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" Nov 02 17:52:34.219145 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:34.219120 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" Nov 02 17:52:34.484462 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:52:34.484385 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:52:34.507143 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:34.507114 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd"] Nov 02 17:52:35.389334 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:35.389294 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" event={"ID":"5e5069cc-7c3d-4f35-8a37-c874e0a834f3","Type":"ContainerStarted","Data":"179b56e8bdfd3aa8d9a28ae10b59d097d2c7bd68dad8a1aeaa7b7ddac8a4ec4c"} Nov 02 17:52:35.389864 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:35.389402 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" Nov 02 17:52:35.389864 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:35.389429 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" event={"ID":"5e5069cc-7c3d-4f35-8a37-c874e0a834f3","Type":"ContainerStarted","Data":"fa20aae5a31da66cc6c5731458ff657eba1968b1e7bbf8dbf68bb17355d0771d"} Nov 02 17:52:35.410609 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:35.410564 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" podStartSLOduration=2.410552197 podStartE2EDuration="2.410552197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:35.406525306 +0000 UTC m=+1120.097579318" watchObservedRunningTime="2026-11-02 17:52:35.410552197 +0000 UTC m=+1120.101606209" Nov 02 17:52:38.638291 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:38.638244 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r"] Nov 02 17:52:38.642303 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:38.642278 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" Nov 02 17:52:38.647466 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:38.647437 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-eda8e2-serving-cert\"" Nov 02 17:52:38.648096 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:38.648064 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fda100c-a25e-454a-9a8f-044d4a07fa81-metrics-certs\") pod \"maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r\" (UID: \"0fda100c-a25e-454a-9a8f-044d4a07fa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" Nov 02 17:52:38.648254 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:38.648231 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tjb6\" (UniqueName: \"kubernetes.io/projected/0fda100c-a25e-454a-9a8f-044d4a07fa81-kube-api-access-9tjb6\") pod \"maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r\" (UID: \"0fda100c-a25e-454a-9a8f-044d4a07fa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" Nov 02 17:52:38.648500 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:38.648476 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fda100c-a25e-454a-9a8f-044d4a07fa81-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r\" (UID: \"0fda100c-a25e-454a-9a8f-044d4a07fa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" Nov 02 17:52:38.648653 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:38.648537 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fda100c-a25e-454a-9a8f-044d4a07fa81-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r\" (UID: \"0fda100c-a25e-454a-9a8f-044d4a07fa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" Nov 02 17:52:38.652758 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:38.652394 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r"] Nov 02 17:52:38.750221 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:38.750185 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tjb6\" (UniqueName: \"kubernetes.io/projected/0fda100c-a25e-454a-9a8f-044d4a07fa81-kube-api-access-9tjb6\") pod \"maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r\" (UID: \"0fda100c-a25e-454a-9a8f-044d4a07fa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" Nov 02 17:52:38.750426 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:38.750243 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fda100c-a25e-454a-9a8f-044d4a07fa81-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r\" (UID: \"0fda100c-a25e-454a-9a8f-044d4a07fa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" Nov 02 17:52:38.750426 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:38.750296 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fda100c-a25e-454a-9a8f-044d4a07fa81-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r\" (UID: \"0fda100c-a25e-454a-9a8f-044d4a07fa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" Nov 02 17:52:38.750566 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:38.750469 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fda100c-a25e-454a-9a8f-044d4a07fa81-metrics-certs\") pod \"maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r\" (UID: \"0fda100c-a25e-454a-9a8f-044d4a07fa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" Nov 02 17:52:38.750668 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:52:38.750647 5601 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-serving-cert: secret "maas-api-e2e-ipp-b-eda8e2-serving-cert" not found Nov 02 17:52:38.750740 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:52:38.750718 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fda100c-a25e-454a-9a8f-044d4a07fa81-maas-api-tls podName:0fda100c-a25e-454a-9a8f-044d4a07fa81 nodeName:}" failed. No retries permitted until 2026-11-02 17:52:39.250695715 +0000 UTC m=+1123.941749722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0fda100c-a25e-454a-9a8f-044d4a07fa81-maas-api-tls") pod "maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" (UID: "0fda100c-a25e-454a-9a8f-044d4a07fa81") : secret "maas-api-e2e-ipp-b-eda8e2-serving-cert" not found Nov 02 17:52:38.761665 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:38.751529 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fda100c-a25e-454a-9a8f-044d4a07fa81-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r\" (UID: \"0fda100c-a25e-454a-9a8f-044d4a07fa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" Nov 02 17:52:38.762265 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:38.762232 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tjb6\" (UniqueName: \"kubernetes.io/projected/0fda100c-a25e-454a-9a8f-044d4a07fa81-kube-api-access-9tjb6\") pod \"maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r\" (UID: \"0fda100c-a25e-454a-9a8f-044d4a07fa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" Nov 02 17:52:38.765542 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:38.765513 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fda100c-a25e-454a-9a8f-044d4a07fa81-metrics-certs\") pod \"maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r\" (UID: \"0fda100c-a25e-454a-9a8f-044d4a07fa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" Nov 02 17:52:39.264566 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:39.264514 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fda100c-a25e-454a-9a8f-044d4a07fa81-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r\" (UID: \"0fda100c-a25e-454a-9a8f-044d4a07fa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" Nov 02 17:52:39.301638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:39.301406 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fda100c-a25e-454a-9a8f-044d4a07fa81-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r\" (UID: \"0fda100c-a25e-454a-9a8f-044d4a07fa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" Nov 02 17:52:39.558156 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:39.558064 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" Nov 02 17:52:39.848409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:39.848376 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r"] Nov 02 17:52:39.848824 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:52:39.848647 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fda100c_a25e_454a_9a8f_044d4a07fa81.slice/crio-8413e378fac7d482eb282ec10d405831c86876f1c41bc17feea2da38caccc298 WatchSource:0}: Error finding container 8413e378fac7d482eb282ec10d405831c86876f1c41bc17feea2da38caccc298: Status 404 returned error can't find the container with id 8413e378fac7d482eb282ec10d405831c86876f1c41bc17feea2da38caccc298 Nov 02 17:52:39.850060 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:39.850037 5601 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:52:40.446384 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:40.446345 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" event={"ID":"0fda100c-a25e-454a-9a8f-044d4a07fa81","Type":"ContainerStarted","Data":"0204e0ddc4a30c569f4189088bf883cf34e613ae2d01fd0674ffcd211e1c5a9a"} Nov 02 17:52:40.446539 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:40.446407 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" event={"ID":"0fda100c-a25e-454a-9a8f-044d4a07fa81","Type":"ContainerStarted","Data":"8413e378fac7d482eb282ec10d405831c86876f1c41bc17feea2da38caccc298"} Nov 02 17:52:40.446539 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:40.446432 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" Nov 02 17:52:40.470752 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:40.470701 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" podStartSLOduration=2.085917025 podStartE2EDuration="2.470684859s" totalImagesPullingTime="384.767834ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:40.465788271 +0000 UTC m=+1125.156842283" watchObservedRunningTime="2026-11-02 17:52:40.470684859 +0000 UTC m=+1125.161738873" Nov 02 17:52:41.830888 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:41.830852 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt"] Nov 02 17:52:41.831378 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:41.831101 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" podUID="466397d6-4f0f-4b1c-98d3-93903456336a" containerName="maas-api" containerID="cri-o://76dda50a601e35ce19d89bd0f2f088a6cefacc223206c307d43ff6d6c868ec8d" gracePeriod=30 Nov 02 17:52:42.036628 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.036581 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs"] Nov 02 17:52:42.171150 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.171121 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" Nov 02 17:52:42.244757 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.244718 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/466397d6-4f0f-4b1c-98d3-93903456336a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466397d6-4f0f-4b1c-98d3-93903456336a-openshift-service-ca\") pod \"466397d6-4f0f-4b1c-98d3-93903456336a\" (UID: \"466397d6-4f0f-4b1c-98d3-93903456336a\") " Nov 02 17:52:42.244959 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.244765 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/466397d6-4f0f-4b1c-98d3-93903456336a-kube-api-access-wbfr6\" (UniqueName: \"kubernetes.io/projected/466397d6-4f0f-4b1c-98d3-93903456336a-kube-api-access-wbfr6\") pod \"466397d6-4f0f-4b1c-98d3-93903456336a\" (UID: \"466397d6-4f0f-4b1c-98d3-93903456336a\") " Nov 02 17:52:42.244959 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.244789 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/466397d6-4f0f-4b1c-98d3-93903456336a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466397d6-4f0f-4b1c-98d3-93903456336a-maas-api-tls\") pod \"466397d6-4f0f-4b1c-98d3-93903456336a\" (UID: \"466397d6-4f0f-4b1c-98d3-93903456336a\") " Nov 02 17:52:42.245082 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.245067 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/466397d6-4f0f-4b1c-98d3-93903456336a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/466397d6-4f0f-4b1c-98d3-93903456336a-metrics-certs\") pod \"466397d6-4f0f-4b1c-98d3-93903456336a\" (UID: \"466397d6-4f0f-4b1c-98d3-93903456336a\") " Nov 02 17:52:42.245191 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.245162 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/466397d6-4f0f-4b1c-98d3-93903456336a-openshift-service-ca" pod "466397d6-4f0f-4b1c-98d3-93903456336a" (UID: "466397d6-4f0f-4b1c-98d3-93903456336a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:42.245703 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.245675 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466397d6-4f0f-4b1c-98d3-93903456336a-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:42.247853 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.247827 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/466397d6-4f0f-4b1c-98d3-93903456336a-metrics-certs" pod "466397d6-4f0f-4b1c-98d3-93903456336a" (UID: "466397d6-4f0f-4b1c-98d3-93903456336a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:52:42.248803 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.248775 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/466397d6-4f0f-4b1c-98d3-93903456336a-maas-api-tls" pod "466397d6-4f0f-4b1c-98d3-93903456336a" (UID: "466397d6-4f0f-4b1c-98d3-93903456336a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:52:42.248895 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.248826 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466397d6-4f0f-4b1c-98d3-93903456336a-kube-api-access-wbfr6" pod "466397d6-4f0f-4b1c-98d3-93903456336a" (UID: "466397d6-4f0f-4b1c-98d3-93903456336a"). InnerVolumeSpecName "kube-api-access-wbfr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:42.346567 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.346531 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbfr6\" (UniqueName: \"kubernetes.io/projected/466397d6-4f0f-4b1c-98d3-93903456336a-kube-api-access-wbfr6\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:42.346567 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.346565 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466397d6-4f0f-4b1c-98d3-93903456336a-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:42.346780 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.346578 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/466397d6-4f0f-4b1c-98d3-93903456336a-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:42.530643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.521546 5601 generic.go:356] "Generic (PLEG): container finished" podID="466397d6-4f0f-4b1c-98d3-93903456336a" containerID="76dda50a601e35ce19d89bd0f2f088a6cefacc223206c307d43ff6d6c868ec8d" exitCode=0 Nov 02 17:52:42.530643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.521602 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" event={"ID":"466397d6-4f0f-4b1c-98d3-93903456336a","Type":"ContainerDied","Data":"76dda50a601e35ce19d89bd0f2f088a6cefacc223206c307d43ff6d6c868ec8d"} Nov 02 17:52:42.530643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.521686 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" event={"ID":"466397d6-4f0f-4b1c-98d3-93903456336a","Type":"ContainerDied","Data":"1b6cfc2d5ab95908677f0ad44a95a640ba565cfda5abeb1821a6e5821709acaf"} Nov 02 17:52:42.530643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.521721 5601 scope.go:122] "RemoveContainer" containerID="76dda50a601e35ce19d89bd0f2f088a6cefacc223206c307d43ff6d6c868ec8d" Nov 02 17:52:42.530643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.523260 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt" Nov 02 17:52:42.609259 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.609230 5601 scope.go:122] "RemoveContainer" containerID="76dda50a601e35ce19d89bd0f2f088a6cefacc223206c307d43ff6d6c868ec8d" Nov 02 17:52:42.610490 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:52:42.610444 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76dda50a601e35ce19d89bd0f2f088a6cefacc223206c307d43ff6d6c868ec8d\": container with ID starting with 76dda50a601e35ce19d89bd0f2f088a6cefacc223206c307d43ff6d6c868ec8d not found: ID does not exist" containerID="76dda50a601e35ce19d89bd0f2f088a6cefacc223206c307d43ff6d6c868ec8d" Nov 02 17:52:42.610605 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.610503 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76dda50a601e35ce19d89bd0f2f088a6cefacc223206c307d43ff6d6c868ec8d"} err="failed to get container status \"76dda50a601e35ce19d89bd0f2f088a6cefacc223206c307d43ff6d6c868ec8d\": rpc error: code = NotFound desc = could not find container \"76dda50a601e35ce19d89bd0f2f088a6cefacc223206c307d43ff6d6c868ec8d\": container with ID starting with 76dda50a601e35ce19d89bd0f2f088a6cefacc223206c307d43ff6d6c868ec8d not found: ID does not exist" Nov 02 17:52:42.643568 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.643530 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt"] Nov 02 17:52:42.646645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:42.646608 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e4fdc1-5fb4d58555-cdkpt"] Nov 02 17:52:43.427269 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:43.427226 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="466397d6-4f0f-4b1c-98d3-93903456336a" path="/var/lib/kubelet/pods/466397d6-4f0f-4b1c-98d3-93903456336a/volumes" Nov 02 17:52:44.339641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.330690 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh"] Nov 02 17:52:44.339641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.331587 5601 cpu_manager.go:414] "containerMap: removing container" podUID="466397d6-4f0f-4b1c-98d3-93903456336a" containerName="maas-api" Nov 02 17:52:44.339641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.331608 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="466397d6-4f0f-4b1c-98d3-93903456336a" containerName="maas-api" Nov 02 17:52:44.339641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.331872 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="466397d6-4f0f-4b1c-98d3-93903456336a" containerName="maas-api" Nov 02 17:52:44.339641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.336322 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" Nov 02 17:52:44.349375 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.349349 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-d8e217-serving-cert\"" Nov 02 17:52:44.352970 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.352943 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh"] Nov 02 17:52:44.451640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.450182 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/583bc588-e41e-4af3-802f-fec734c1cfd1-metrics-certs\") pod \"maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh\" (UID: \"583bc588-e41e-4af3-802f-fec734c1cfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" Nov 02 17:52:44.451640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.450322 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtbpn\" (UniqueName: \"kubernetes.io/projected/583bc588-e41e-4af3-802f-fec734c1cfd1-kube-api-access-jtbpn\") pod \"maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh\" (UID: \"583bc588-e41e-4af3-802f-fec734c1cfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" Nov 02 17:52:44.451640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.450391 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/583bc588-e41e-4af3-802f-fec734c1cfd1-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh\" (UID: \"583bc588-e41e-4af3-802f-fec734c1cfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" Nov 02 17:52:44.451640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.450456 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/583bc588-e41e-4af3-802f-fec734c1cfd1-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh\" (UID: \"583bc588-e41e-4af3-802f-fec734c1cfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" Nov 02 17:52:44.556638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.555662 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/583bc588-e41e-4af3-802f-fec734c1cfd1-metrics-certs\") pod \"maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh\" (UID: \"583bc588-e41e-4af3-802f-fec734c1cfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" Nov 02 17:52:44.556638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.555873 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtbpn\" (UniqueName: \"kubernetes.io/projected/583bc588-e41e-4af3-802f-fec734c1cfd1-kube-api-access-jtbpn\") pod \"maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh\" (UID: \"583bc588-e41e-4af3-802f-fec734c1cfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" Nov 02 17:52:44.556638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.555978 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/583bc588-e41e-4af3-802f-fec734c1cfd1-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh\" (UID: \"583bc588-e41e-4af3-802f-fec734c1cfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" Nov 02 17:52:44.556638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.556059 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/583bc588-e41e-4af3-802f-fec734c1cfd1-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh\" (UID: \"583bc588-e41e-4af3-802f-fec734c1cfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" Nov 02 17:52:44.570494 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.570417 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/583bc588-e41e-4af3-802f-fec734c1cfd1-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh\" (UID: \"583bc588-e41e-4af3-802f-fec734c1cfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" Nov 02 17:52:44.589272 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.589235 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/583bc588-e41e-4af3-802f-fec734c1cfd1-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh\" (UID: \"583bc588-e41e-4af3-802f-fec734c1cfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" Nov 02 17:52:44.614385 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.614275 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtbpn\" (UniqueName: \"kubernetes.io/projected/583bc588-e41e-4af3-802f-fec734c1cfd1-kube-api-access-jtbpn\") pod \"maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh\" (UID: \"583bc588-e41e-4af3-802f-fec734c1cfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" Nov 02 17:52:44.644636 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.641249 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/583bc588-e41e-4af3-802f-fec734c1cfd1-metrics-certs\") pod \"maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh\" (UID: \"583bc588-e41e-4af3-802f-fec734c1cfd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" Nov 02 17:52:44.691139 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:44.690866 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" Nov 02 17:52:45.156024 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:45.151555 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh"] Nov 02 17:52:45.201246 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:52:45.201214 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583bc588_e41e_4af3_802f_fec734c1cfd1.slice/crio-a36e0b62792e17eeb51189e6e4fffd48cfbd92b038a5d00dfa47625b73b16458 WatchSource:0}: Error finding container a36e0b62792e17eeb51189e6e4fffd48cfbd92b038a5d00dfa47625b73b16458: Status 404 returned error can't find the container with id a36e0b62792e17eeb51189e6e4fffd48cfbd92b038a5d00dfa47625b73b16458 Nov 02 17:52:45.652742 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:45.652697 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" event={"ID":"583bc588-e41e-4af3-802f-fec734c1cfd1","Type":"ContainerStarted","Data":"a36e0b62792e17eeb51189e6e4fffd48cfbd92b038a5d00dfa47625b73b16458"} Nov 02 17:52:46.422524 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:46.422497 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" Nov 02 17:52:46.472330 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:46.472296 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" Nov 02 17:52:46.667050 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:46.667007 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" event={"ID":"583bc588-e41e-4af3-802f-fec734c1cfd1","Type":"ContainerStarted","Data":"bd4224af7bc57ec10c5cea9f86f510ac728571c9403819692ddc05a1113cd39c"} Nov 02 17:52:46.667501 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:46.667180 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" Nov 02 17:52:46.691421 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:46.691308 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" podStartSLOduration=2.141061201 podStartE2EDuration="2.691294263s" totalImagesPullingTime="550.233062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:46.689208623 +0000 UTC m=+1131.380262628" watchObservedRunningTime="2026-11-02 17:52:46.691294263 +0000 UTC m=+1131.382348275" Nov 02 17:52:47.046693 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:47.037680 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs" podUID="cdb5b8e3-8e9c-463c-b122-1724581fdd8f" containerName="payload-pre-processing" containerID="cri-o://20075eadfd26461bd6f4c536a0149b40eca237c541706364b8c817a89b22f43b" gracePeriod=25 Nov 02 17:52:47.401808 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:47.401782 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs" Nov 02 17:52:47.533666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:47.533581 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb5b8e3-8e9c-463c-b122-1724581fdd8f-kube-api-access-p29nx\" (UniqueName: \"kubernetes.io/projected/cdb5b8e3-8e9c-463c-b122-1724581fdd8f-kube-api-access-p29nx\") pod \"cdb5b8e3-8e9c-463c-b122-1724581fdd8f\" (UID: \"cdb5b8e3-8e9c-463c-b122-1724581fdd8f\") " Nov 02 17:52:47.533864 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:47.533802 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdb5b8e3-8e9c-463c-b122-1724581fdd8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb5b8e3-8e9c-463c-b122-1724581fdd8f-plugins-config-volume\") pod \"cdb5b8e3-8e9c-463c-b122-1724581fdd8f\" (UID: \"cdb5b8e3-8e9c-463c-b122-1724581fdd8f\") " Nov 02 17:52:47.556141 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:47.556088 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb5b8e3-8e9c-463c-b122-1724581fdd8f-kube-api-access-p29nx" pod "cdb5b8e3-8e9c-463c-b122-1724581fdd8f" (UID: "cdb5b8e3-8e9c-463c-b122-1724581fdd8f"). InnerVolumeSpecName "kube-api-access-p29nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:47.560846 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:47.560814 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb5b8e3-8e9c-463c-b122-1724581fdd8f-plugins-config-volume" pod "cdb5b8e3-8e9c-463c-b122-1724581fdd8f" (UID: "cdb5b8e3-8e9c-463c-b122-1724581fdd8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:47.653699 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:47.652797 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p29nx\" (UniqueName: \"kubernetes.io/projected/cdb5b8e3-8e9c-463c-b122-1724581fdd8f-kube-api-access-p29nx\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:47.653699 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:47.652836 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb5b8e3-8e9c-463c-b122-1724581fdd8f-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:47.699487 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:47.699446 5601 generic.go:356] "Generic (PLEG): container finished" podID="cdb5b8e3-8e9c-463c-b122-1724581fdd8f" containerID="20075eadfd26461bd6f4c536a0149b40eca237c541706364b8c817a89b22f43b" exitCode=0 Nov 02 17:52:47.700672 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:47.700367 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs" Nov 02 17:52:47.702871 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:47.702839 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs" event={"ID":"cdb5b8e3-8e9c-463c-b122-1724581fdd8f","Type":"ContainerDied","Data":"20075eadfd26461bd6f4c536a0149b40eca237c541706364b8c817a89b22f43b"} Nov 02 17:52:47.703040 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:47.702902 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs" event={"ID":"cdb5b8e3-8e9c-463c-b122-1724581fdd8f","Type":"ContainerDied","Data":"dfd82dd07f05c5621d6f98aeeca013b4be1bb1b68837d0702da6acf3468e5845"} Nov 02 17:52:47.703040 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:47.702930 5601 scope.go:122] "RemoveContainer" containerID="20075eadfd26461bd6f4c536a0149b40eca237c541706364b8c817a89b22f43b" Nov 02 17:52:47.721790 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:47.721750 5601 scope.go:122] "RemoveContainer" containerID="20075eadfd26461bd6f4c536a0149b40eca237c541706364b8c817a89b22f43b" Nov 02 17:52:47.722878 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:52:47.722848 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20075eadfd26461bd6f4c536a0149b40eca237c541706364b8c817a89b22f43b\": container with ID starting with 20075eadfd26461bd6f4c536a0149b40eca237c541706364b8c817a89b22f43b not found: ID does not exist" containerID="20075eadfd26461bd6f4c536a0149b40eca237c541706364b8c817a89b22f43b" Nov 02 17:52:47.722994 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:47.722921 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20075eadfd26461bd6f4c536a0149b40eca237c541706364b8c817a89b22f43b"} err="failed to get container status \"20075eadfd26461bd6f4c536a0149b40eca237c541706364b8c817a89b22f43b\": rpc error: code = NotFound desc = could not find container \"20075eadfd26461bd6f4c536a0149b40eca237c541706364b8c817a89b22f43b\": container with ID starting with 20075eadfd26461bd6f4c536a0149b40eca237c541706364b8c817a89b22f43b not found: ID does not exist" Nov 02 17:52:47.776754 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:47.776713 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs"] Nov 02 17:52:47.779845 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:47.779807 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7dbdb8-7b497bf779-x4hzs"] Nov 02 17:52:49.427281 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:49.427247 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdb5b8e3-8e9c-463c-b122-1724581fdd8f" path="/var/lib/kubelet/pods/cdb5b8e3-8e9c-463c-b122-1724581fdd8f/volumes" Nov 02 17:52:52.708117 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:52.708083 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" Nov 02 17:52:54.762753 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:54.762722 5601 generic.go:356] "Generic (PLEG): container finished" podID="9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" containerID="f207fee6549bcc23658d0a936a3137680497dc455e647eee3b28470c1ba95e9d" exitCode=0 Nov 02 17:52:54.763191 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:54.762794 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" event={"ID":"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40","Type":"ContainerDied","Data":"f207fee6549bcc23658d0a936a3137680497dc455e647eee3b28470c1ba95e9d"} Nov 02 17:52:54.763251 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:54.763195 5601 scope.go:122] "RemoveContainer" containerID="f207fee6549bcc23658d0a936a3137680497dc455e647eee3b28470c1ba95e9d" Nov 02 17:52:54.764668 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:54.764607 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:52:54.764784 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:54.764721 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:52:54.764848 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:54.764793 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:52:55.772692 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:55.772658 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" event={"ID":"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40","Type":"ContainerStarted","Data":"e99a4a7e5fff9454525da8a13200aeb7f9a4fd25c3fe2f362470aa29c144fdcb"} Nov 02 17:52:55.977844 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:55.977813 5601 scope.go:122] "RemoveContainer" containerID="4dc8ac7a3054619e9036846edbf077daab0d8aa87061a7552e1ba57ff52abeb1" Nov 02 17:52:56.690661 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:56.690624 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:52:56.692293 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:56.692260 5601 patch_prober.go:29] interesting pod/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 17:52:56.692437 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:56.692330 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" podUID="9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 17:52:57.690367 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:57.690328 5601 patch_prober.go:29] interesting pod/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 17:52:57.690986 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:57.690390 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" podUID="9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 17:52:58.690250 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:58.690211 5601 patch_prober.go:29] interesting pod/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 17:52:58.690436 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:58.690267 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" podUID="9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 17:52:59.690395 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:59.690359 5601 patch_prober.go:29] interesting pod/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 17:52:59.690597 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:52:59.690418 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" podUID="9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 17:53:00.690413 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:00.690372 5601 patch_prober.go:29] interesting pod/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 17:53:00.690582 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:00.690441 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" podUID="9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 17:53:00.948070 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:00.947964 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:53:01.694978 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:01.694951 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:53:01.870010 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:01.869983 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:53:04.497302 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:53:04.497260 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:53:22.045667 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:22.045632 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84"] Nov 02 17:53:22.046281 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:22.045898 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" podUID="9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" containerName="istio-proxy" containerID="cri-o://e99a4a7e5fff9454525da8a13200aeb7f9a4fd25c3fe2f362470aa29c144fdcb" gracePeriod=30 Nov 02 17:53:27.345558 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.345534 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:53:27.410771 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.410743 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-data\") pod \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " Nov 02 17:53:27.410945 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.410831 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-envoy\") pod \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " Nov 02 17:53:27.410945 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.410870 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istiod-ca-cert\") pod \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " Nov 02 17:53:27.410945 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.410925 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-kube-api-access-2xx2n\" (UniqueName: \"kubernetes.io/projected/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-kube-api-access-2xx2n\") pod \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " Nov 02 17:53:27.411110 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.410998 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-token\" (UniqueName: \"kubernetes.io/projected/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-token\") pod \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " Nov 02 17:53:27.411110 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.411025 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-data" pod "9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" (UID: "9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:27.411110 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.411082 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-credential-socket\") pod \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " Nov 02 17:53:27.411269 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.411136 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-workload-certs\") pod \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " Nov 02 17:53:27.411269 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.411232 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-podinfo\") pod \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " Nov 02 17:53:27.411371 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.411266 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-workload-socket\") pod \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\" (UID: \"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40\") " Nov 02 17:53:27.411448 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.411419 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-credential-socket" pod "9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" (UID: "9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:27.412034 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.411598 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-workload-certs" pod "9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" (UID: "9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:27.412034 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.411821 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-workload-socket" pod "9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" (UID: "9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:27.412034 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.411997 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istiod-ca-cert" pod "9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" (UID: "9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:27.412301 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.412156 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:27.412301 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.412195 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:27.412301 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.412209 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:27.412301 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.412223 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:27.412301 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.412236 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:27.413760 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.413721 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-token" pod "9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" (UID: "9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:27.414004 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.413978 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-envoy" pod "9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" (UID: "9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:27.414209 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.414186 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-podinfo" pod "9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" (UID: "9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:53:27.414278 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.414242 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-kube-api-access-2xx2n" pod "9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" (UID: "9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40"). InnerVolumeSpecName "kube-api-access-2xx2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:27.513727 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.513701 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:27.513727 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.513725 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xx2n\" (UniqueName: \"kubernetes.io/projected/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-kube-api-access-2xx2n\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:27.513892 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.513734 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:27.513892 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:27.513743 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:28.111976 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:28.111943 5601 generic.go:356] "Generic (PLEG): container finished" podID="9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" containerID="e99a4a7e5fff9454525da8a13200aeb7f9a4fd25c3fe2f362470aa29c144fdcb" exitCode=0 Nov 02 17:53:28.112165 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:28.111996 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" Nov 02 17:53:28.112165 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:28.112026 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" event={"ID":"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40","Type":"ContainerDied","Data":"e99a4a7e5fff9454525da8a13200aeb7f9a4fd25c3fe2f362470aa29c144fdcb"} Nov 02 17:53:28.112165 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:28.112090 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84" event={"ID":"9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40","Type":"ContainerDied","Data":"8c673a43ad3ad02a235185bb113a86c2b8c73d8ce490a39c228fbca3411a120b"} Nov 02 17:53:28.112165 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:28.112121 5601 scope.go:122] "RemoveContainer" containerID="e99a4a7e5fff9454525da8a13200aeb7f9a4fd25c3fe2f362470aa29c144fdcb" Nov 02 17:53:28.122565 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:28.122547 5601 scope.go:122] "RemoveContainer" containerID="f207fee6549bcc23658d0a936a3137680497dc455e647eee3b28470c1ba95e9d" Nov 02 17:53:28.144663 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:28.144583 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84"] Nov 02 17:53:28.154072 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:28.153984 5601 scope.go:122] "RemoveContainer" containerID="e99a4a7e5fff9454525da8a13200aeb7f9a4fd25c3fe2f362470aa29c144fdcb" Nov 02 17:53:28.154918 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:53:28.154741 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e99a4a7e5fff9454525da8a13200aeb7f9a4fd25c3fe2f362470aa29c144fdcb\": container with ID starting with e99a4a7e5fff9454525da8a13200aeb7f9a4fd25c3fe2f362470aa29c144fdcb not found: ID does not exist" containerID="e99a4a7e5fff9454525da8a13200aeb7f9a4fd25c3fe2f362470aa29c144fdcb" Nov 02 17:53:28.154918 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:28.154782 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e99a4a7e5fff9454525da8a13200aeb7f9a4fd25c3fe2f362470aa29c144fdcb"} err="failed to get container status \"e99a4a7e5fff9454525da8a13200aeb7f9a4fd25c3fe2f362470aa29c144fdcb\": rpc error: code = NotFound desc = could not find container \"e99a4a7e5fff9454525da8a13200aeb7f9a4fd25c3fe2f362470aa29c144fdcb\": container with ID starting with e99a4a7e5fff9454525da8a13200aeb7f9a4fd25c3fe2f362470aa29c144fdcb not found: ID does not exist" Nov 02 17:53:28.154918 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:28.154808 5601 scope.go:122] "RemoveContainer" containerID="f207fee6549bcc23658d0a936a3137680497dc455e647eee3b28470c1ba95e9d" Nov 02 17:53:28.155480 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:53:28.155412 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f207fee6549bcc23658d0a936a3137680497dc455e647eee3b28470c1ba95e9d\": container with ID starting with f207fee6549bcc23658d0a936a3137680497dc455e647eee3b28470c1ba95e9d not found: ID does not exist" containerID="f207fee6549bcc23658d0a936a3137680497dc455e647eee3b28470c1ba95e9d" Nov 02 17:53:28.155480 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:28.155444 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f207fee6549bcc23658d0a936a3137680497dc455e647eee3b28470c1ba95e9d"} err="failed to get container status \"f207fee6549bcc23658d0a936a3137680497dc455e647eee3b28470c1ba95e9d\": rpc error: code = NotFound desc = could not find container \"f207fee6549bcc23658d0a936a3137680497dc455e647eee3b28470c1ba95e9d\": container with ID starting with f207fee6549bcc23658d0a936a3137680497dc455e647eee3b28470c1ba95e9d not found: ID does not exist" Nov 02 17:53:28.156635 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:28.156596 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-4a5f09-openshift-default-854695dbf7-6tt84"] Nov 02 17:53:29.427086 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:29.427057 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" path="/var/lib/kubelet/pods/9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40/volumes" Nov 02 17:53:30.267154 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:30.267118 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq"] Nov 02 17:53:30.267458 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:30.267408 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" podUID="ffeee2eb-8d38-40dd-b000-4863eed2bd86" containerName="maas-api" containerID="cri-o://abcfc78a15c15b93ae709a0a158d842d21636e8d3ccbd1004238fe1ded18d7bf" gracePeriod=30 Nov 02 17:53:30.614648 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:30.614593 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" Nov 02 17:53:30.684868 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:30.682766 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffeee2eb-8d38-40dd-b000-4863eed2bd86-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffeee2eb-8d38-40dd-b000-4863eed2bd86-maas-api-tls\") pod \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\" (UID: \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\") " Nov 02 17:53:30.684868 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:30.682857 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffeee2eb-8d38-40dd-b000-4863eed2bd86-kube-api-access-sxwwm\" (UniqueName: \"kubernetes.io/projected/ffeee2eb-8d38-40dd-b000-4863eed2bd86-kube-api-access-sxwwm\") pod \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\" (UID: \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\") " Nov 02 17:53:30.684868 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:30.682935 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffeee2eb-8d38-40dd-b000-4863eed2bd86-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffeee2eb-8d38-40dd-b000-4863eed2bd86-metrics-certs\") pod \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\" (UID: \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\") " Nov 02 17:53:30.684868 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:30.683138 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffeee2eb-8d38-40dd-b000-4863eed2bd86-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffeee2eb-8d38-40dd-b000-4863eed2bd86-openshift-service-ca\") pod \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\" (UID: \"ffeee2eb-8d38-40dd-b000-4863eed2bd86\") " Nov 02 17:53:30.691198 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:30.687950 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffeee2eb-8d38-40dd-b000-4863eed2bd86-openshift-service-ca" pod "ffeee2eb-8d38-40dd-b000-4863eed2bd86" (UID: "ffeee2eb-8d38-40dd-b000-4863eed2bd86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:30.730653 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:30.729773 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffeee2eb-8d38-40dd-b000-4863eed2bd86-metrics-certs" pod "ffeee2eb-8d38-40dd-b000-4863eed2bd86" (UID: "ffeee2eb-8d38-40dd-b000-4863eed2bd86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:30.770918 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:30.764847 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffeee2eb-8d38-40dd-b000-4863eed2bd86-maas-api-tls" pod "ffeee2eb-8d38-40dd-b000-4863eed2bd86" (UID: "ffeee2eb-8d38-40dd-b000-4863eed2bd86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:30.778173 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:30.777329 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffeee2eb-8d38-40dd-b000-4863eed2bd86-kube-api-access-sxwwm" pod "ffeee2eb-8d38-40dd-b000-4863eed2bd86" (UID: "ffeee2eb-8d38-40dd-b000-4863eed2bd86"). InnerVolumeSpecName "kube-api-access-sxwwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:30.794141 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:30.794101 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffeee2eb-8d38-40dd-b000-4863eed2bd86-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:30.794141 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:30.794138 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffeee2eb-8d38-40dd-b000-4863eed2bd86-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:30.794349 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:30.794152 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxwwm\" (UniqueName: \"kubernetes.io/projected/ffeee2eb-8d38-40dd-b000-4863eed2bd86-kube-api-access-sxwwm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:30.794349 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:30.794165 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffeee2eb-8d38-40dd-b000-4863eed2bd86-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:31.136791 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:31.136750 5601 generic.go:356] "Generic (PLEG): container finished" podID="ffeee2eb-8d38-40dd-b000-4863eed2bd86" containerID="abcfc78a15c15b93ae709a0a158d842d21636e8d3ccbd1004238fe1ded18d7bf" exitCode=0 Nov 02 17:53:31.137003 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:31.136803 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" Nov 02 17:53:31.137003 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:31.136830 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" event={"ID":"ffeee2eb-8d38-40dd-b000-4863eed2bd86","Type":"ContainerDied","Data":"abcfc78a15c15b93ae709a0a158d842d21636e8d3ccbd1004238fe1ded18d7bf"} Nov 02 17:53:31.137003 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:31.136898 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq" event={"ID":"ffeee2eb-8d38-40dd-b000-4863eed2bd86","Type":"ContainerDied","Data":"4465311a8f761d50ae727a97407e4926ae824725f1adbb067e58ddfde13b3756"} Nov 02 17:53:31.137003 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:31.136928 5601 scope.go:122] "RemoveContainer" containerID="abcfc78a15c15b93ae709a0a158d842d21636e8d3ccbd1004238fe1ded18d7bf" Nov 02 17:53:31.148726 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:31.148706 5601 scope.go:122] "RemoveContainer" containerID="abcfc78a15c15b93ae709a0a158d842d21636e8d3ccbd1004238fe1ded18d7bf" Nov 02 17:53:31.149006 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:53:31.148987 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abcfc78a15c15b93ae709a0a158d842d21636e8d3ccbd1004238fe1ded18d7bf\": container with ID starting with abcfc78a15c15b93ae709a0a158d842d21636e8d3ccbd1004238fe1ded18d7bf not found: ID does not exist" containerID="abcfc78a15c15b93ae709a0a158d842d21636e8d3ccbd1004238fe1ded18d7bf" Nov 02 17:53:31.149054 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:31.149014 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abcfc78a15c15b93ae709a0a158d842d21636e8d3ccbd1004238fe1ded18d7bf"} err="failed to get container status \"abcfc78a15c15b93ae709a0a158d842d21636e8d3ccbd1004238fe1ded18d7bf\": rpc error: code = NotFound desc = could not find container \"abcfc78a15c15b93ae709a0a158d842d21636e8d3ccbd1004238fe1ded18d7bf\": container with ID starting with abcfc78a15c15b93ae709a0a158d842d21636e8d3ccbd1004238fe1ded18d7bf not found: ID does not exist" Nov 02 17:53:31.168913 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:31.168883 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq"] Nov 02 17:53:31.188347 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:31.188313 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6098d0-6499599c65-pddfq"] Nov 02 17:53:31.427251 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:31.427220 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffeee2eb-8d38-40dd-b000-4863eed2bd86" path="/var/lib/kubelet/pods/ffeee2eb-8d38-40dd-b000-4863eed2bd86/volumes" Nov 02 17:53:34.485082 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:53:34.485044 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:53:43.377725 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:43.377642 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265"] Nov 02 17:53:43.378143 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:43.377869 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" podUID="3465ffa7-2dbd-41c8-99e6-3e484be099d7" containerName="istio-proxy" containerID="cri-o://4dffe2ea87bd1dcb9c8b148a1de4d31f84dc783a884db3aba19c7edaea1ef949" gracePeriod=30 Nov 02 17:53:49.624077 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.624033 5601 generic.go:356] "Generic (PLEG): container finished" podID="3465ffa7-2dbd-41c8-99e6-3e484be099d7" containerID="4dffe2ea87bd1dcb9c8b148a1de4d31f84dc783a884db3aba19c7edaea1ef949" exitCode=0 Nov 02 17:53:49.624693 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.624667 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" event={"ID":"3465ffa7-2dbd-41c8-99e6-3e484be099d7","Type":"ContainerDied","Data":"4dffe2ea87bd1dcb9c8b148a1de4d31f84dc783a884db3aba19c7edaea1ef949"} Nov 02 17:53:49.624910 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.624892 5601 scope.go:122] "RemoveContainer" containerID="caa28cabe1402d18625dd9d3dbe64e91775577eed6ef6635c6ed4a914028f554" Nov 02 17:53:49.663180 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.662823 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:53:49.852723 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.845447 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-podinfo\") pod \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " Nov 02 17:53:49.852723 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.845514 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-workload-socket\") pod \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " Nov 02 17:53:49.852723 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.845640 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-envoy\") pod \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " Nov 02 17:53:49.854568 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.854245 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-workload-certs" pod "3465ffa7-2dbd-41c8-99e6-3e484be099d7" (UID: "3465ffa7-2dbd-41c8-99e6-3e484be099d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:49.866894 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.866860 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-workload-socket" pod "3465ffa7-2dbd-41c8-99e6-3e484be099d7" (UID: "3465ffa7-2dbd-41c8-99e6-3e484be099d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:49.908202 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.906675 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-workload-certs\") pod \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " Nov 02 17:53:49.908202 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.906830 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istiod-ca-cert\") pod \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " Nov 02 17:53:49.908202 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.906884 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3465ffa7-2dbd-41c8-99e6-3e484be099d7-kube-api-access-dktl2\" (UniqueName: \"kubernetes.io/projected/3465ffa7-2dbd-41c8-99e6-3e484be099d7-kube-api-access-dktl2\") pod \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " Nov 02 17:53:49.908202 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.906958 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-data\") pod \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " Nov 02 17:53:49.908202 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.907047 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-credential-socket\") pod \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " Nov 02 17:53:49.908202 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.907086 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-token\" (UniqueName: \"kubernetes.io/projected/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-token\") pod \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\" (UID: \"3465ffa7-2dbd-41c8-99e6-3e484be099d7\") " Nov 02 17:53:49.908202 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.907775 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:49.908202 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.907801 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:49.955052 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.954937 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-credential-socket" pod "3465ffa7-2dbd-41c8-99e6-3e484be099d7" (UID: "3465ffa7-2dbd-41c8-99e6-3e484be099d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:49.955052 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.955022 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-data" pod "3465ffa7-2dbd-41c8-99e6-3e484be099d7" (UID: "3465ffa7-2dbd-41c8-99e6-3e484be099d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:49.970975 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.970876 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istiod-ca-cert" pod "3465ffa7-2dbd-41c8-99e6-3e484be099d7" (UID: "3465ffa7-2dbd-41c8-99e6-3e484be099d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:49.997520 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.984802 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-podinfo" pod "3465ffa7-2dbd-41c8-99e6-3e484be099d7" (UID: "3465ffa7-2dbd-41c8-99e6-3e484be099d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:53:49.997520 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.984897 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-envoy" pod "3465ffa7-2dbd-41c8-99e6-3e484be099d7" (UID: "3465ffa7-2dbd-41c8-99e6-3e484be099d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:49.997520 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.996723 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:49.999334 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:49.998027 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-token" pod "3465ffa7-2dbd-41c8-99e6-3e484be099d7" (UID: "3465ffa7-2dbd-41c8-99e6-3e484be099d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:50.000639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:50.000579 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3465ffa7-2dbd-41c8-99e6-3e484be099d7-kube-api-access-dktl2" pod "3465ffa7-2dbd-41c8-99e6-3e484be099d7" (UID: "3465ffa7-2dbd-41c8-99e6-3e484be099d7"). InnerVolumeSpecName "kube-api-access-dktl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:50.015358 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:50.015208 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:50.015358 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:50.015250 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dktl2\" (UniqueName: \"kubernetes.io/projected/3465ffa7-2dbd-41c8-99e6-3e484be099d7-kube-api-access-dktl2\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:50.015358 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:50.015271 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:50.015358 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:50.015286 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:50.015358 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:50.015299 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:50.015358 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:50.015313 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:50.015358 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:50.015328 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3465ffa7-2dbd-41c8-99e6-3e484be099d7-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:50.100077 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:50.100032 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:50.140717 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:50.140594 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:50.144007 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:50.143971 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:50.509638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:50.506859 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:50.659663 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:50.656175 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" event={"ID":"3465ffa7-2dbd-41c8-99e6-3e484be099d7","Type":"ContainerDied","Data":"58732885cf782bd09e0b26e35ee4e2eaf3dfe7e4e1eba5ad84f99d268cfed1c4"} Nov 02 17:53:50.659663 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:50.656252 5601 scope.go:122] "RemoveContainer" containerID="4dffe2ea87bd1dcb9c8b148a1de4d31f84dc783a884db3aba19c7edaea1ef949" Nov 02 17:53:50.659663 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:50.656376 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265" Nov 02 17:53:50.815700 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:50.813671 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265"] Nov 02 17:53:50.855643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:50.844064 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6098d0-openshift-default-5ccb49c75c-js265"] Nov 02 17:53:51.498508 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:51.498467 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3465ffa7-2dbd-41c8-99e6-3e484be099d7" path="/var/lib/kubelet/pods/3465ffa7-2dbd-41c8-99e6-3e484be099d7/volumes" Nov 02 17:53:54.346563 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.346522 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r"] Nov 02 17:53:54.348480 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.348439 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" podUID="0fda100c-a25e-454a-9a8f-044d4a07fa81" containerName="maas-api" containerID="cri-o://0204e0ddc4a30c569f4189088bf883cf34e613ae2d01fd0674ffcd211e1c5a9a" gracePeriod=30 Nov 02 17:53:54.696409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.696176 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" Nov 02 17:53:54.710951 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.710912 5601 generic.go:356] "Generic (PLEG): container finished" podID="0fda100c-a25e-454a-9a8f-044d4a07fa81" containerID="0204e0ddc4a30c569f4189088bf883cf34e613ae2d01fd0674ffcd211e1c5a9a" exitCode=0 Nov 02 17:53:54.711263 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.711241 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" event={"ID":"0fda100c-a25e-454a-9a8f-044d4a07fa81","Type":"ContainerDied","Data":"0204e0ddc4a30c569f4189088bf883cf34e613ae2d01fd0674ffcd211e1c5a9a"} Nov 02 17:53:54.711370 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.711304 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" event={"ID":"0fda100c-a25e-454a-9a8f-044d4a07fa81","Type":"ContainerDied","Data":"8413e378fac7d482eb282ec10d405831c86876f1c41bc17feea2da38caccc298"} Nov 02 17:53:54.711370 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.711339 5601 scope.go:122] "RemoveContainer" containerID="0204e0ddc4a30c569f4189088bf883cf34e613ae2d01fd0674ffcd211e1c5a9a" Nov 02 17:53:54.711843 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.711488 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r" Nov 02 17:53:54.777700 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.774019 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fda100c-a25e-454a-9a8f-044d4a07fa81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fda100c-a25e-454a-9a8f-044d4a07fa81-openshift-service-ca\") pod \"0fda100c-a25e-454a-9a8f-044d4a07fa81\" (UID: \"0fda100c-a25e-454a-9a8f-044d4a07fa81\") " Nov 02 17:53:54.777700 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.774160 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fda100c-a25e-454a-9a8f-044d4a07fa81-kube-api-access-9tjb6\" (UniqueName: \"kubernetes.io/projected/0fda100c-a25e-454a-9a8f-044d4a07fa81-kube-api-access-9tjb6\") pod \"0fda100c-a25e-454a-9a8f-044d4a07fa81\" (UID: \"0fda100c-a25e-454a-9a8f-044d4a07fa81\") " Nov 02 17:53:54.777700 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.774199 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fda100c-a25e-454a-9a8f-044d4a07fa81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fda100c-a25e-454a-9a8f-044d4a07fa81-metrics-certs\") pod \"0fda100c-a25e-454a-9a8f-044d4a07fa81\" (UID: \"0fda100c-a25e-454a-9a8f-044d4a07fa81\") " Nov 02 17:53:54.777700 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.774254 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fda100c-a25e-454a-9a8f-044d4a07fa81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fda100c-a25e-454a-9a8f-044d4a07fa81-maas-api-tls\") pod \"0fda100c-a25e-454a-9a8f-044d4a07fa81\" (UID: \"0fda100c-a25e-454a-9a8f-044d4a07fa81\") " Nov 02 17:53:54.777700 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.777576 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fda100c-a25e-454a-9a8f-044d4a07fa81-openshift-service-ca" pod "0fda100c-a25e-454a-9a8f-044d4a07fa81" (UID: "0fda100c-a25e-454a-9a8f-044d4a07fa81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:54.837927 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.837864 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fda100c-a25e-454a-9a8f-044d4a07fa81-kube-api-access-9tjb6" pod "0fda100c-a25e-454a-9a8f-044d4a07fa81" (UID: "0fda100c-a25e-454a-9a8f-044d4a07fa81"). InnerVolumeSpecName "kube-api-access-9tjb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:54.875638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.874800 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fda100c-a25e-454a-9a8f-044d4a07fa81-metrics-certs" pod "0fda100c-a25e-454a-9a8f-044d4a07fa81" (UID: "0fda100c-a25e-454a-9a8f-044d4a07fa81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:54.875638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.874898 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fda100c-a25e-454a-9a8f-044d4a07fa81-maas-api-tls" pod "0fda100c-a25e-454a-9a8f-044d4a07fa81" (UID: "0fda100c-a25e-454a-9a8f-044d4a07fa81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:54.885640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.876574 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fda100c-a25e-454a-9a8f-044d4a07fa81-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:54.885640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.876609 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tjb6\" (UniqueName: \"kubernetes.io/projected/0fda100c-a25e-454a-9a8f-044d4a07fa81-kube-api-access-9tjb6\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:54.885640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.876653 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fda100c-a25e-454a-9a8f-044d4a07fa81-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:54.885640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.876669 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fda100c-a25e-454a-9a8f-044d4a07fa81-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:54.895320 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.892797 5601 scope.go:122] "RemoveContainer" containerID="0204e0ddc4a30c569f4189088bf883cf34e613ae2d01fd0674ffcd211e1c5a9a" Nov 02 17:53:54.895320 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:53:54.893983 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0204e0ddc4a30c569f4189088bf883cf34e613ae2d01fd0674ffcd211e1c5a9a\": container with ID starting with 0204e0ddc4a30c569f4189088bf883cf34e613ae2d01fd0674ffcd211e1c5a9a not found: ID does not exist" containerID="0204e0ddc4a30c569f4189088bf883cf34e613ae2d01fd0674ffcd211e1c5a9a" Nov 02 17:53:54.895320 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:54.894015 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0204e0ddc4a30c569f4189088bf883cf34e613ae2d01fd0674ffcd211e1c5a9a"} err="failed to get container status \"0204e0ddc4a30c569f4189088bf883cf34e613ae2d01fd0674ffcd211e1c5a9a\": rpc error: code = NotFound desc = could not find container \"0204e0ddc4a30c569f4189088bf883cf34e613ae2d01fd0674ffcd211e1c5a9a\": container with ID starting with 0204e0ddc4a30c569f4189088bf883cf34e613ae2d01fd0674ffcd211e1c5a9a not found: ID does not exist" Nov 02 17:53:55.043019 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:55.042988 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r"] Nov 02 17:53:55.054510 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:55.054479 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda8e2-c6df98d86-tgk2r"] Nov 02 17:53:55.432024 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:55.431984 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fda100c-a25e-454a-9a8f-044d4a07fa81" path="/var/lib/kubelet/pods/0fda100c-a25e-454a-9a8f-044d4a07fa81/volumes" Nov 02 17:53:56.025075 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:53:56.025037 5601 scope.go:122] "RemoveContainer" containerID="076593f510bf4a6e9d432857ee8fafdf3a3895a5c4dbf4680b8b56cacdf04b39" Nov 02 17:54:04.485218 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:54:04.485180 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:54:27.594466 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.594431 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484"] Nov 02 17:54:27.595135 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.595058 5601 cpu_manager.go:414] "containerMap: removing container" podUID="9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" containerName="istio-proxy" Nov 02 17:54:27.595135 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.595073 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" containerName="istio-proxy" Nov 02 17:54:27.595135 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.595083 5601 cpu_manager.go:414] "containerMap: removing container" podUID="cdb5b8e3-8e9c-463c-b122-1724581fdd8f" containerName="payload-pre-processing" Nov 02 17:54:27.595135 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.595088 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdb5b8e3-8e9c-463c-b122-1724581fdd8f" containerName="payload-pre-processing" Nov 02 17:54:27.595135 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.595100 5601 cpu_manager.go:414] "containerMap: removing container" podUID="ffeee2eb-8d38-40dd-b000-4863eed2bd86" containerName="maas-api" Nov 02 17:54:27.595135 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.595105 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffeee2eb-8d38-40dd-b000-4863eed2bd86" containerName="maas-api" Nov 02 17:54:27.595135 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.595127 5601 cpu_manager.go:414] "containerMap: removing container" podUID="0fda100c-a25e-454a-9a8f-044d4a07fa81" containerName="maas-api" Nov 02 17:54:27.595135 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.595133 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fda100c-a25e-454a-9a8f-044d4a07fa81" containerName="maas-api" Nov 02 17:54:27.595571 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.595145 5601 cpu_manager.go:414] "containerMap: removing container" podUID="3465ffa7-2dbd-41c8-99e6-3e484be099d7" containerName="istio-proxy" Nov 02 17:54:27.595571 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.595150 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3465ffa7-2dbd-41c8-99e6-3e484be099d7" containerName="istio-proxy" Nov 02 17:54:27.595571 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.595163 5601 cpu_manager.go:414] "containerMap: removing container" podUID="3465ffa7-2dbd-41c8-99e6-3e484be099d7" containerName="istio-proxy" Nov 02 17:54:27.595571 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.595168 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3465ffa7-2dbd-41c8-99e6-3e484be099d7" containerName="istio-proxy" Nov 02 17:54:27.595571 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.595275 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" containerName="istio-proxy" Nov 02 17:54:27.595571 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.595283 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdb5b8e3-8e9c-463c-b122-1724581fdd8f" containerName="payload-pre-processing" Nov 02 17:54:27.595571 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.595291 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffeee2eb-8d38-40dd-b000-4863eed2bd86" containerName="maas-api" Nov 02 17:54:27.595571 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.595305 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fda100c-a25e-454a-9a8f-044d4a07fa81" containerName="maas-api" Nov 02 17:54:27.595571 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.595313 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="3465ffa7-2dbd-41c8-99e6-3e484be099d7" containerName="istio-proxy" Nov 02 17:54:27.599053 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.599033 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.607264 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.607230 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-56e6db-openshift-default-dockercfg-hc54x\"" Nov 02 17:54:27.615171 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.615146 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484"] Nov 02 17:54:27.654051 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.654012 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48837eb5-5658-4f69-894b-d1075a574e20-istio-podinfo\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.654233 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.654061 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-istio-envoy\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.654233 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.654155 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-workload-certs\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.654233 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.654200 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-workload-socket\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.654409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.654325 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-credential-socket\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.654505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.654486 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znzj2\" (UniqueName: \"kubernetes.io/projected/48837eb5-5658-4f69-894b-d1075a574e20-kube-api-access-znzj2\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.654566 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.654522 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-istio-data\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.654650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.654609 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48837eb5-5658-4f69-894b-d1075a574e20-istiod-ca-cert\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.654716 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.654685 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48837eb5-5658-4f69-894b-d1075a574e20-istio-token\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.763640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.763479 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48837eb5-5658-4f69-894b-d1075a574e20-istio-token\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.763640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.763526 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48837eb5-5658-4f69-894b-d1075a574e20-istio-podinfo\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.763640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.763579 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-istio-envoy\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.771673 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.764108 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-workload-certs\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.771673 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.764464 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-workload-certs\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.771673 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.764558 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-workload-socket\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.771673 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.764714 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-credential-socket\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.771673 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.764840 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znzj2\" (UniqueName: \"kubernetes.io/projected/48837eb5-5658-4f69-894b-d1075a574e20-kube-api-access-znzj2\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.771673 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.764871 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-istio-data\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.771673 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.764963 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48837eb5-5658-4f69-894b-d1075a574e20-istiod-ca-cert\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.771673 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.765460 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-workload-socket\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.771673 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.765741 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-credential-socket\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.771673 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.766252 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-istio-data\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.771673 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.766717 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48837eb5-5658-4f69-894b-d1075a574e20-istiod-ca-cert\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.779640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.779279 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48837eb5-5658-4f69-894b-d1075a574e20-istio-podinfo\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.806277 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.801463 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znzj2\" (UniqueName: \"kubernetes.io/projected/48837eb5-5658-4f69-894b-d1075a574e20-kube-api-access-znzj2\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.806277 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.802107 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-istio-envoy\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.824642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.823099 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48837eb5-5658-4f69-894b-d1075a574e20-istio-token\") pod \"e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:27.914637 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:27.914524 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:28.227566 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:28.227371 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484"] Nov 02 17:54:28.232892 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:28.232846 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:54:28.233020 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:28.232942 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:54:28.233020 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:28.232987 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:54:29.046725 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:29.046686 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" event={"ID":"48837eb5-5658-4f69-894b-d1075a574e20","Type":"ContainerStarted","Data":"eece78161e35b2ac0fe9c5be28dfa91d594c11b119ba0bf55a81832a06f7aa6f"} Nov 02 17:54:29.047144 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:29.046756 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" event={"ID":"48837eb5-5658-4f69-894b-d1075a574e20","Type":"ContainerStarted","Data":"0fd1c6ea0bf0f18aad9675678520cc0c4b31b1d7727bf5f8ace9e1961ae07121"} Nov 02 17:54:29.072472 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:29.072331 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" podStartSLOduration=2.072311489 podStartE2EDuration="2.072311489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:29.071822149 +0000 UTC m=+1233.762876163" watchObservedRunningTime="2026-11-02 17:54:29.072311489 +0000 UTC m=+1233.763365501" Nov 02 17:54:29.915603 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:29.915567 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:29.921327 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:29.921299 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:30.053801 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:30.053767 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:30.054916 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:54:30.054893 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:54:34.486988 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:54:34.486949 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:55:04.485866 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:55:04.485829 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:55:12.545061 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:12.545012 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:12.567877 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:12.567831 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 17:55:12.803908 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:12.796870 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:12.875751 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:12.875703 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:13.004686 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:12.999186 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:13.185647 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:13.184732 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:21.671222 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:21.671189 5601 generic.go:356] "Generic (PLEG): container finished" podID="48837eb5-5658-4f69-894b-d1075a574e20" containerID="eece78161e35b2ac0fe9c5be28dfa91d594c11b119ba0bf55a81832a06f7aa6f" exitCode=0 Nov 02 17:55:21.671782 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:21.671271 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" event={"ID":"48837eb5-5658-4f69-894b-d1075a574e20","Type":"ContainerDied","Data":"eece78161e35b2ac0fe9c5be28dfa91d594c11b119ba0bf55a81832a06f7aa6f"} Nov 02 17:55:21.671782 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:21.671736 5601 scope.go:122] "RemoveContainer" containerID="eece78161e35b2ac0fe9c5be28dfa91d594c11b119ba0bf55a81832a06f7aa6f" Nov 02 17:55:21.673309 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:21.673269 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:55:21.673375 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:21.673342 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:55:21.673411 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:21.673378 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:55:22.682456 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:22.682422 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" event={"ID":"48837eb5-5658-4f69-894b-d1075a574e20","Type":"ContainerStarted","Data":"dbc44c1435ceaba72f2565a0c6f66544d6b372d519333fe898592faa70431013"} Nov 02 17:55:22.915528 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:22.915493 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:55:22.915822 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:22.915772 5601 patch_prober.go:29] interesting pod/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:55:22.915929 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:22.915891 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" podUID="48837eb5-5658-4f69-894b-d1075a574e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 17:55:23.915281 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:23.915237 5601 patch_prober.go:29] interesting pod/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:55:23.915771 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:23.915306 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" podUID="48837eb5-5658-4f69-894b-d1075a574e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 17:55:24.915131 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:24.915094 5601 patch_prober.go:29] interesting pod/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:55:24.915344 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:24.915157 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" podUID="48837eb5-5658-4f69-894b-d1075a574e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 17:55:25.918078 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:25.918040 5601 patch_prober.go:29] interesting pod/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:55:25.922751 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:25.922701 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" podUID="48837eb5-5658-4f69-894b-d1075a574e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 17:55:26.876642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:26.872036 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:27.004641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:26.928606 5601 patch_prober.go:29] interesting pod/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:55:27.004641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:27.000953 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" podUID="48837eb5-5658-4f69-894b-d1075a574e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 17:55:27.011792 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:27.011737 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Nov 02 17:55:27.114770 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:27.114708 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:27.147260 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:27.147098 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:27.162759 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:27.162706 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 17:55:27.467692 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:27.467645 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:27.556980 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:27.555741 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:27.594839 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:27.594776 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:27.796699 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:27.794707 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 17:55:27.887647 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:27.885031 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:27.887647 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:27.885846 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:55:27.932651 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:27.931186 5601 patch_prober.go:29] interesting pod/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:55:27.932651 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:27.931245 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" podUID="48837eb5-5658-4f69-894b-d1075a574e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 17:55:27.932651 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:27.931673 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:28.021986 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:28.021938 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:28.033642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:28.023104 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:55:28.115955 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:28.115860 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 02 17:55:28.116838 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:28.116811 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:55:28.141283 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:28.127603 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:28.154835 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:28.154795 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 02 17:55:28.171071 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:28.168861 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:28.171608 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:28.171584 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:55:28.171824 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:28.171807 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:55:28.297484 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:28.297453 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:55:28.409835 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:28.409188 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:55:28.424793 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:28.424766 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:55:28.478094 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:28.477059 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:55:28.480762 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:28.480702 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:55:29.240315 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:29.240242 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:55:29.241059 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:29.240777 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:55:29.242933 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:29.242914 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:55:34.485591 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:55:34.485552 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:55:38.257669 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.251958 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9"] Nov 02 17:55:38.257669 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.252884 5601 cpu_manager.go:414] "containerMap: removing container" podUID="9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" containerName="istio-proxy" Nov 02 17:55:38.257669 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.252901 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" containerName="istio-proxy" Nov 02 17:55:38.257669 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.253056 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="3465ffa7-2dbd-41c8-99e6-3e484be099d7" containerName="istio-proxy" Nov 02 17:55:38.257669 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.253076 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7e67db-b4ab-496e-b7f0-d1af7f4c7b40" containerName="istio-proxy" Nov 02 17:55:38.257669 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.255733 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.351175 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.351134 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9"] Nov 02 17:55:38.374523 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.374478 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648e1353-2923-484a-835e-90b96e1acc50-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.374846 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.374823 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648e1353-2923-484a-835e-90b96e1acc50-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.375060 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.375041 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.375207 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.375189 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7lp7\" (UniqueName: \"kubernetes.io/projected/648e1353-2923-484a-835e-90b96e1acc50-kube-api-access-q7lp7\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.375388 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.375361 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.375832 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.375810 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.375988 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.375970 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.376108 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.376093 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.376484 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.376461 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648e1353-2923-484a-835e-90b96e1acc50-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.485639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.484931 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648e1353-2923-484a-835e-90b96e1acc50-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.485639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.485002 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648e1353-2923-484a-835e-90b96e1acc50-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.485639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.485124 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.485639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.485161 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7lp7\" (UniqueName: \"kubernetes.io/projected/648e1353-2923-484a-835e-90b96e1acc50-kube-api-access-q7lp7\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.485639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.485194 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.485639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.485237 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.485639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.485264 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.485639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.485293 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.485639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.485562 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648e1353-2923-484a-835e-90b96e1acc50-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.509137 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.488737 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648e1353-2923-484a-835e-90b96e1acc50-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.509137 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.495492 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.509137 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.495787 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.509137 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.501962 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.536640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.510421 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.547695 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.539002 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648e1353-2923-484a-835e-90b96e1acc50-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.564636 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.563976 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7lp7\" (UniqueName: \"kubernetes.io/projected/648e1353-2923-484a-835e-90b96e1acc50-kube-api-access-q7lp7\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.588376 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.585606 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648e1353-2923-484a-835e-90b96e1acc50-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.590374 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.590351 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:38.878287 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:38.878191 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:39.294948 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:39.292366 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9"] Nov 02 17:55:39.302562 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:39.301313 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:55:39.302562 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:39.301406 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:55:39.302562 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:39.301453 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:55:39.331992 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:39.331961 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" event={"ID":"648e1353-2923-484a-835e-90b96e1acc50","Type":"ContainerStarted","Data":"80fef00ff834393adae309e695d29ef74d259fc29e4a6d520eebbf890a2c3f63"} Nov 02 17:55:40.365806 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:40.365761 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" event={"ID":"648e1353-2923-484a-835e-90b96e1acc50","Type":"ContainerStarted","Data":"2efea605288b4f806b659f00a712e7b83375159b042e45742587baf15c7c800a"} Nov 02 17:55:40.395659 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:40.395594 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" podStartSLOduration=2.395580926 podStartE2EDuration="2.395580926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:40.390956494 +0000 UTC m=+1305.082010507" watchObservedRunningTime="2026-11-02 17:55:40.395580926 +0000 UTC m=+1305.086634937" Nov 02 17:55:40.879450 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:40.879411 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:40.881091 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:40.881067 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 02 17:55:40.881177 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:40.881127 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" podUID="648e1353-2923-484a-835e-90b96e1acc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 02 17:55:41.879728 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:41.879694 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 02 17:55:41.880164 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:41.879742 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" podUID="648e1353-2923-484a-835e-90b96e1acc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 02 17:55:42.879319 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:42.879279 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 02 17:55:42.879507 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:42.879340 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" podUID="648e1353-2923-484a-835e-90b96e1acc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 02 17:55:43.879609 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:43.879559 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 02 17:55:43.880006 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:43.879637 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" podUID="648e1353-2923-484a-835e-90b96e1acc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 02 17:55:44.881652 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:44.881405 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 02 17:55:44.881652 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:44.881454 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" podUID="648e1353-2923-484a-835e-90b96e1acc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 02 17:55:45.879687 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:45.879180 5601 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 02 17:55:45.879687 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:45.879250 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" podUID="648e1353-2923-484a-835e-90b96e1acc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 02 17:55:46.972468 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:46.972433 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:46.973130 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:46.973110 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:46.974512 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:46.974486 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 17:55:47.054427 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:47.054384 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484"] Nov 02 17:55:47.055229 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:47.054930 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" podUID="48837eb5-5658-4f69-894b-d1075a574e20" containerName="istio-proxy" containerID="cri-o://dbc44c1435ceaba72f2565a0c6f66544d6b372d519333fe898592faa70431013" gracePeriod=30 Nov 02 17:55:52.329653 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.329560 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:55:52.415307 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.415271 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48837eb5-5658-4f69-894b-d1075a574e20-kube-api-access-znzj2\" (UniqueName: \"kubernetes.io/projected/48837eb5-5658-4f69-894b-d1075a574e20-kube-api-access-znzj2\") pod \"48837eb5-5658-4f69-894b-d1075a574e20\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " Nov 02 17:55:52.415475 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.415365 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-istio-envoy\") pod \"48837eb5-5658-4f69-894b-d1075a574e20\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " Nov 02 17:55:52.415475 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.415412 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-workload-socket\") pod \"48837eb5-5658-4f69-894b-d1075a574e20\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " Nov 02 17:55:52.415547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.415484 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48837eb5-5658-4f69-894b-d1075a574e20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48837eb5-5658-4f69-894b-d1075a574e20-istio-podinfo\") pod \"48837eb5-5658-4f69-894b-d1075a574e20\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " Nov 02 17:55:52.415594 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.415556 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48837eb5-5658-4f69-894b-d1075a574e20-istio-token\" (UniqueName: \"kubernetes.io/projected/48837eb5-5658-4f69-894b-d1075a574e20-istio-token\") pod \"48837eb5-5658-4f69-894b-d1075a574e20\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " Nov 02 17:55:52.415730 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.415708 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-credential-socket\") pod \"48837eb5-5658-4f69-894b-d1075a574e20\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " Nov 02 17:55:52.415802 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.415771 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48837eb5-5658-4f69-894b-d1075a574e20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48837eb5-5658-4f69-894b-d1075a574e20-istiod-ca-cert\") pod \"48837eb5-5658-4f69-894b-d1075a574e20\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " Nov 02 17:55:52.415802 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.415785 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-workload-socket" pod "48837eb5-5658-4f69-894b-d1075a574e20" (UID: "48837eb5-5658-4f69-894b-d1075a574e20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:52.415917 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.415818 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-istio-data\") pod \"48837eb5-5658-4f69-894b-d1075a574e20\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " Nov 02 17:55:52.415972 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.415918 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-workload-certs\") pod \"48837eb5-5658-4f69-894b-d1075a574e20\" (UID: \"48837eb5-5658-4f69-894b-d1075a574e20\") " Nov 02 17:55:52.416033 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.415999 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-credential-socket" pod "48837eb5-5658-4f69-894b-d1075a574e20" (UID: "48837eb5-5658-4f69-894b-d1075a574e20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:52.416323 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.416275 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-workload-certs" pod "48837eb5-5658-4f69-894b-d1075a574e20" (UID: "48837eb5-5658-4f69-894b-d1075a574e20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:52.416421 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.416350 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-istio-data" pod "48837eb5-5658-4f69-894b-d1075a574e20" (UID: "48837eb5-5658-4f69-894b-d1075a574e20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:52.416634 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.416590 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:52.416634 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.416631 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:52.416853 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.416647 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:52.416853 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.416661 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:52.416853 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.416827 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48837eb5-5658-4f69-894b-d1075a574e20-istiod-ca-cert" pod "48837eb5-5658-4f69-894b-d1075a574e20" (UID: "48837eb5-5658-4f69-894b-d1075a574e20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:52.417909 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.417888 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-istio-envoy" pod "48837eb5-5658-4f69-894b-d1075a574e20" (UID: "48837eb5-5658-4f69-894b-d1075a574e20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:52.418087 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.418068 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48837eb5-5658-4f69-894b-d1075a574e20-kube-api-access-znzj2" pod "48837eb5-5658-4f69-894b-d1075a574e20" (UID: "48837eb5-5658-4f69-894b-d1075a574e20"). InnerVolumeSpecName "kube-api-access-znzj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:52.418156 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.418098 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48837eb5-5658-4f69-894b-d1075a574e20-istio-token" pod "48837eb5-5658-4f69-894b-d1075a574e20" (UID: "48837eb5-5658-4f69-894b-d1075a574e20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:52.418216 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.418194 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48837eb5-5658-4f69-894b-d1075a574e20-istio-podinfo" pod "48837eb5-5658-4f69-894b-d1075a574e20" (UID: "48837eb5-5658-4f69-894b-d1075a574e20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:55:52.475275 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.475246 5601 generic.go:356] "Generic (PLEG): container finished" podID="48837eb5-5658-4f69-894b-d1075a574e20" containerID="dbc44c1435ceaba72f2565a0c6f66544d6b372d519333fe898592faa70431013" exitCode=0 Nov 02 17:55:52.475411 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.475297 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" Nov 02 17:55:52.475411 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.475331 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" event={"ID":"48837eb5-5658-4f69-894b-d1075a574e20","Type":"ContainerDied","Data":"dbc44c1435ceaba72f2565a0c6f66544d6b372d519333fe898592faa70431013"} Nov 02 17:55:52.475411 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.475394 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484" event={"ID":"48837eb5-5658-4f69-894b-d1075a574e20","Type":"ContainerDied","Data":"0fd1c6ea0bf0f18aad9675678520cc0c4b31b1d7727bf5f8ace9e1961ae07121"} Nov 02 17:55:52.475558 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.475417 5601 scope.go:122] "RemoveContainer" containerID="dbc44c1435ceaba72f2565a0c6f66544d6b372d519333fe898592faa70431013" Nov 02 17:55:52.486448 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.486428 5601 scope.go:122] "RemoveContainer" containerID="eece78161e35b2ac0fe9c5be28dfa91d594c11b119ba0bf55a81832a06f7aa6f" Nov 02 17:55:52.509139 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.509112 5601 scope.go:122] "RemoveContainer" containerID="dbc44c1435ceaba72f2565a0c6f66544d6b372d519333fe898592faa70431013" Nov 02 17:55:52.509520 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:55:52.509487 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc44c1435ceaba72f2565a0c6f66544d6b372d519333fe898592faa70431013\": container with ID starting with dbc44c1435ceaba72f2565a0c6f66544d6b372d519333fe898592faa70431013 not found: ID does not exist" containerID="dbc44c1435ceaba72f2565a0c6f66544d6b372d519333fe898592faa70431013" Nov 02 17:55:52.509665 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.509532 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc44c1435ceaba72f2565a0c6f66544d6b372d519333fe898592faa70431013"} err="failed to get container status \"dbc44c1435ceaba72f2565a0c6f66544d6b372d519333fe898592faa70431013\": rpc error: code = NotFound desc = could not find container \"dbc44c1435ceaba72f2565a0c6f66544d6b372d519333fe898592faa70431013\": container with ID starting with dbc44c1435ceaba72f2565a0c6f66544d6b372d519333fe898592faa70431013 not found: ID does not exist" Nov 02 17:55:52.509665 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.509558 5601 scope.go:122] "RemoveContainer" containerID="eece78161e35b2ac0fe9c5be28dfa91d594c11b119ba0bf55a81832a06f7aa6f" Nov 02 17:55:52.510507 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:55:52.510479 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eece78161e35b2ac0fe9c5be28dfa91d594c11b119ba0bf55a81832a06f7aa6f\": container with ID starting with eece78161e35b2ac0fe9c5be28dfa91d594c11b119ba0bf55a81832a06f7aa6f not found: ID does not exist" containerID="eece78161e35b2ac0fe9c5be28dfa91d594c11b119ba0bf55a81832a06f7aa6f" Nov 02 17:55:52.510609 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.510512 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eece78161e35b2ac0fe9c5be28dfa91d594c11b119ba0bf55a81832a06f7aa6f"} err="failed to get container status \"eece78161e35b2ac0fe9c5be28dfa91d594c11b119ba0bf55a81832a06f7aa6f\": rpc error: code = NotFound desc = could not find container \"eece78161e35b2ac0fe9c5be28dfa91d594c11b119ba0bf55a81832a06f7aa6f\": container with ID starting with eece78161e35b2ac0fe9c5be28dfa91d594c11b119ba0bf55a81832a06f7aa6f not found: ID does not exist" Nov 02 17:55:52.511079 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.510738 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484"] Nov 02 17:55:52.513755 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.513730 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-56e6db-openshift-default-bf78764b5-8t484"] Nov 02 17:55:52.518395 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.518373 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znzj2\" (UniqueName: \"kubernetes.io/projected/48837eb5-5658-4f69-894b-d1075a574e20-kube-api-access-znzj2\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:52.518484 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.518397 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48837eb5-5658-4f69-894b-d1075a574e20-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:52.518484 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.518413 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48837eb5-5658-4f69-894b-d1075a574e20-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:52.518484 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.518425 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48837eb5-5658-4f69-894b-d1075a574e20-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:52.518484 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:52.518438 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48837eb5-5658-4f69-894b-d1075a574e20-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:53.429035 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:55:53.429003 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48837eb5-5658-4f69-894b-d1075a574e20" path="/var/lib/kubelet/pods/48837eb5-5658-4f69-894b-d1075a574e20/volumes" Nov 02 17:56:00.530003 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:00.529964 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd"] Nov 02 17:56:04.512853 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:56:04.512381 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:56:05.531221 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:05.531182 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" podUID="5e5069cc-7c3d-4f35-8a37-c874e0a834f3" containerName="payload-pre-processing" containerID="cri-o://179b56e8bdfd3aa8d9a28ae10b59d097d2c7bd68dad8a1aeaa7b7ddac8a4ec4c" gracePeriod=25 Nov 02 17:56:06.421803 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:06.421766 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" podUID="5e5069cc-7c3d-4f35-8a37-c874e0a834f3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:56:07.780497 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:07.780455 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc"] Nov 02 17:56:07.781341 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:07.781318 5601 cpu_manager.go:414] "containerMap: removing container" podUID="48837eb5-5658-4f69-894b-d1075a574e20" containerName="istio-proxy" Nov 02 17:56:07.781387 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:07.781345 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48837eb5-5658-4f69-894b-d1075a574e20" containerName="istio-proxy" Nov 02 17:56:07.781387 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:07.781365 5601 cpu_manager.go:414] "containerMap: removing container" podUID="48837eb5-5658-4f69-894b-d1075a574e20" containerName="istio-proxy" Nov 02 17:56:07.781387 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:07.781373 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48837eb5-5658-4f69-894b-d1075a574e20" containerName="istio-proxy" Nov 02 17:56:07.781585 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:07.781570 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="48837eb5-5658-4f69-894b-d1075a574e20" containerName="istio-proxy" Nov 02 17:56:07.781670 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:07.781590 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="48837eb5-5658-4f69-894b-d1075a574e20" containerName="istio-proxy" Nov 02 17:56:07.787391 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:07.787368 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" Nov 02 17:56:07.806082 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:07.806052 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-b6b6f4-serving-cert\"" Nov 02 17:56:07.808390 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:07.808355 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc"] Nov 02 17:56:07.959520 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:07.959477 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98ac353-914e-4f05-b8b5-2c29eb3703d0-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc\" (UID: \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" Nov 02 17:56:07.959766 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:07.959534 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98ac353-914e-4f05-b8b5-2c29eb3703d0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc\" (UID: \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" Nov 02 17:56:07.959766 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:07.959699 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98ac353-914e-4f05-b8b5-2c29eb3703d0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc\" (UID: \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" Nov 02 17:56:07.959904 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:07.959885 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-445gg\" (UniqueName: \"kubernetes.io/projected/b98ac353-914e-4f05-b8b5-2c29eb3703d0-kube-api-access-445gg\") pod \"maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc\" (UID: \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" Nov 02 17:56:08.062758 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:08.062652 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-445gg\" (UniqueName: \"kubernetes.io/projected/b98ac353-914e-4f05-b8b5-2c29eb3703d0-kube-api-access-445gg\") pod \"maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc\" (UID: \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" Nov 02 17:56:08.062941 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:08.062825 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98ac353-914e-4f05-b8b5-2c29eb3703d0-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc\" (UID: \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" Nov 02 17:56:08.062941 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:08.062865 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98ac353-914e-4f05-b8b5-2c29eb3703d0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc\" (UID: \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" Nov 02 17:56:08.062941 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:08.062889 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98ac353-914e-4f05-b8b5-2c29eb3703d0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc\" (UID: \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" Nov 02 17:56:08.064004 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:08.063969 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98ac353-914e-4f05-b8b5-2c29eb3703d0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc\" (UID: \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" Nov 02 17:56:08.072327 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:08.072290 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98ac353-914e-4f05-b8b5-2c29eb3703d0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc\" (UID: \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" Nov 02 17:56:08.082650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:08.082200 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98ac353-914e-4f05-b8b5-2c29eb3703d0-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc\" (UID: \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" Nov 02 17:56:08.091549 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:08.091471 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-445gg\" (UniqueName: \"kubernetes.io/projected/b98ac353-914e-4f05-b8b5-2c29eb3703d0-kube-api-access-445gg\") pod \"maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc\" (UID: \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" Nov 02 17:56:08.130641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:08.120908 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" Nov 02 17:56:08.547833 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:08.547799 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc"] Nov 02 17:56:08.558358 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:56:08.558310 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98ac353_914e_4f05_b8b5_2c29eb3703d0.slice/crio-f393778ba06f734c084fe880fd4128e3bd6dc360fed34a490953574259da4452 WatchSource:0}: Error finding container f393778ba06f734c084fe880fd4128e3bd6dc360fed34a490953574259da4452: Status 404 returned error can't find the container with id f393778ba06f734c084fe880fd4128e3bd6dc360fed34a490953574259da4452 Nov 02 17:56:08.673188 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:08.673155 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" event={"ID":"b98ac353-914e-4f05-b8b5-2c29eb3703d0","Type":"ContainerStarted","Data":"f393778ba06f734c084fe880fd4128e3bd6dc360fed34a490953574259da4452"} Nov 02 17:56:09.683227 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:09.683188 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" event={"ID":"b98ac353-914e-4f05-b8b5-2c29eb3703d0","Type":"ContainerStarted","Data":"1fadfd30e93738c4f3b3e7cbe965ae45ee619224e3dddd65a415e433f11b3080"} Nov 02 17:56:09.683650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:09.683282 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" Nov 02 17:56:09.705939 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:09.705893 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" podStartSLOduration=2.150046775 podStartE2EDuration="2.705878149s" totalImagesPullingTime="555.831374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:09.703548236 +0000 UTC m=+1334.394602251" watchObservedRunningTime="2026-11-02 17:56:09.705878149 +0000 UTC m=+1334.396932160" Nov 02 17:56:15.700409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:15.700381 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" Nov 02 17:56:16.431922 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:16.431882 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" podUID="5e5069cc-7c3d-4f35-8a37-c874e0a834f3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:56:22.271227 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.271182 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh"] Nov 02 17:56:22.271748 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.271448 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" podUID="583bc588-e41e-4af3-802f-fec734c1cfd1" containerName="maas-api" containerID="cri-o://bd4224af7bc57ec10c5cea9f86f510ac728571c9403819692ddc05a1113cd39c" gracePeriod=30 Nov 02 17:56:22.709878 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.709839 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" Nov 02 17:56:22.835980 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.835948 5601 generic.go:356] "Generic (PLEG): container finished" podID="583bc588-e41e-4af3-802f-fec734c1cfd1" containerID="bd4224af7bc57ec10c5cea9f86f510ac728571c9403819692ddc05a1113cd39c" exitCode=0 Nov 02 17:56:22.836160 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.835996 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" Nov 02 17:56:22.836160 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.836034 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" event={"ID":"583bc588-e41e-4af3-802f-fec734c1cfd1","Type":"ContainerDied","Data":"bd4224af7bc57ec10c5cea9f86f510ac728571c9403819692ddc05a1113cd39c"} Nov 02 17:56:22.836160 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.836092 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh" event={"ID":"583bc588-e41e-4af3-802f-fec734c1cfd1","Type":"ContainerDied","Data":"a36e0b62792e17eeb51189e6e4fffd48cfbd92b038a5d00dfa47625b73b16458"} Nov 02 17:56:22.836160 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.836121 5601 scope.go:122] "RemoveContainer" containerID="bd4224af7bc57ec10c5cea9f86f510ac728571c9403819692ddc05a1113cd39c" Nov 02 17:56:22.846599 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.846580 5601 scope.go:122] "RemoveContainer" containerID="bd4224af7bc57ec10c5cea9f86f510ac728571c9403819692ddc05a1113cd39c" Nov 02 17:56:22.846970 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:56:22.846945 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd4224af7bc57ec10c5cea9f86f510ac728571c9403819692ddc05a1113cd39c\": container with ID starting with bd4224af7bc57ec10c5cea9f86f510ac728571c9403819692ddc05a1113cd39c not found: ID does not exist" containerID="bd4224af7bc57ec10c5cea9f86f510ac728571c9403819692ddc05a1113cd39c" Nov 02 17:56:22.847049 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.846981 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd4224af7bc57ec10c5cea9f86f510ac728571c9403819692ddc05a1113cd39c"} err="failed to get container status \"bd4224af7bc57ec10c5cea9f86f510ac728571c9403819692ddc05a1113cd39c\": rpc error: code = NotFound desc = could not find container \"bd4224af7bc57ec10c5cea9f86f510ac728571c9403819692ddc05a1113cd39c\": container with ID starting with bd4224af7bc57ec10c5cea9f86f510ac728571c9403819692ddc05a1113cd39c not found: ID does not exist" Nov 02 17:56:22.851081 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.851054 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583bc588-e41e-4af3-802f-fec734c1cfd1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/583bc588-e41e-4af3-802f-fec734c1cfd1-metrics-certs\") pod \"583bc588-e41e-4af3-802f-fec734c1cfd1\" (UID: \"583bc588-e41e-4af3-802f-fec734c1cfd1\") " Nov 02 17:56:22.851243 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.851119 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583bc588-e41e-4af3-802f-fec734c1cfd1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/583bc588-e41e-4af3-802f-fec734c1cfd1-maas-api-tls\") pod \"583bc588-e41e-4af3-802f-fec734c1cfd1\" (UID: \"583bc588-e41e-4af3-802f-fec734c1cfd1\") " Nov 02 17:56:22.851243 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.851166 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583bc588-e41e-4af3-802f-fec734c1cfd1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/583bc588-e41e-4af3-802f-fec734c1cfd1-openshift-service-ca\") pod \"583bc588-e41e-4af3-802f-fec734c1cfd1\" (UID: \"583bc588-e41e-4af3-802f-fec734c1cfd1\") " Nov 02 17:56:22.851371 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.851350 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583bc588-e41e-4af3-802f-fec734c1cfd1-kube-api-access-jtbpn\" (UniqueName: \"kubernetes.io/projected/583bc588-e41e-4af3-802f-fec734c1cfd1-kube-api-access-jtbpn\") pod \"583bc588-e41e-4af3-802f-fec734c1cfd1\" (UID: \"583bc588-e41e-4af3-802f-fec734c1cfd1\") " Nov 02 17:56:22.851827 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.851801 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583bc588-e41e-4af3-802f-fec734c1cfd1-openshift-service-ca" pod "583bc588-e41e-4af3-802f-fec734c1cfd1" (UID: "583bc588-e41e-4af3-802f-fec734c1cfd1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:22.851981 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.851964 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/583bc588-e41e-4af3-802f-fec734c1cfd1-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:22.853716 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.853609 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583bc588-e41e-4af3-802f-fec734c1cfd1-metrics-certs" pod "583bc588-e41e-4af3-802f-fec734c1cfd1" (UID: "583bc588-e41e-4af3-802f-fec734c1cfd1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:22.854065 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.854037 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583bc588-e41e-4af3-802f-fec734c1cfd1-maas-api-tls" pod "583bc588-e41e-4af3-802f-fec734c1cfd1" (UID: "583bc588-e41e-4af3-802f-fec734c1cfd1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:22.854169 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.854085 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583bc588-e41e-4af3-802f-fec734c1cfd1-kube-api-access-jtbpn" pod "583bc588-e41e-4af3-802f-fec734c1cfd1" (UID: "583bc588-e41e-4af3-802f-fec734c1cfd1"). InnerVolumeSpecName "kube-api-access-jtbpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:22.953510 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.953472 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/583bc588-e41e-4af3-802f-fec734c1cfd1-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:22.953510 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.953504 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/583bc588-e41e-4af3-802f-fec734c1cfd1-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:22.953510 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:22.953514 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtbpn\" (UniqueName: \"kubernetes.io/projected/583bc588-e41e-4af3-802f-fec734c1cfd1-kube-api-access-jtbpn\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:23.151890 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.148995 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl"] Nov 02 17:56:23.151890 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.149955 5601 cpu_manager.go:414] "containerMap: removing container" podUID="583bc588-e41e-4af3-802f-fec734c1cfd1" containerName="maas-api" Nov 02 17:56:23.151890 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.149978 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583bc588-e41e-4af3-802f-fec734c1cfd1" containerName="maas-api" Nov 02 17:56:23.151890 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.150134 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="583bc588-e41e-4af3-802f-fec734c1cfd1" containerName="maas-api" Nov 02 17:56:23.158668 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.155609 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.168414 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.168332 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-2961d5-openshift-default-dockercfg-rqmmk\"" Nov 02 17:56:23.172853 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.172825 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl"] Nov 02 17:56:23.191842 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.191807 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh"] Nov 02 17:56:23.195044 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.195010 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d8e217-7568469db7-b44nh"] Nov 02 17:56:23.258665 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.258545 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-data\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.258853 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.258677 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-workload-certs\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.258853 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.258713 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9jrp\" (UniqueName: \"kubernetes.io/projected/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-kube-api-access-v9jrp\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.258853 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.258758 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-workload-socket\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.259070 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.258913 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-podinfo\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.259070 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.258976 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-envoy\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.259070 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.259026 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-token\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.259070 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.259050 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istiod-ca-cert\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.259281 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.259158 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-credential-socket\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.369655 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.361468 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-podinfo\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.369655 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.361556 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-envoy\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.369655 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.361603 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-token\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.369655 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.361649 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istiod-ca-cert\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.369655 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.361754 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-credential-socket\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.369655 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.361792 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-data\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.369655 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.361831 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-workload-certs\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.369655 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.361861 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9jrp\" (UniqueName: \"kubernetes.io/projected/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-kube-api-access-v9jrp\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.369655 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.361903 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-workload-socket\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.369655 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.362484 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-workload-socket\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.369655 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.363243 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-credential-socket\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.369655 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.363791 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-workload-certs\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.369655 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.364022 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-data\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.380933 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.380854 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istiod-ca-cert\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.396354 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.395011 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-envoy\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.402693 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.400499 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9jrp\" (UniqueName: \"kubernetes.io/projected/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-kube-api-access-v9jrp\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.402693 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.401478 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-podinfo\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.407326 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.407268 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-token\") pod \"e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.455650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.455077 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583bc588-e41e-4af3-802f-fec734c1cfd1" path="/var/lib/kubelet/pods/583bc588-e41e-4af3-802f-fec734c1cfd1/volumes" Nov 02 17:56:23.476038 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.475995 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:23.845448 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.845415 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl"] Nov 02 17:56:23.858351 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.858289 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:56:23.858478 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.858459 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:56:23.858596 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:23.858570 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:56:24.249993 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.249957 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22"] Nov 02 17:56:24.254953 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.254923 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" Nov 02 17:56:24.266144 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.266112 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-9165c3-serving-cert\"" Nov 02 17:56:24.275301 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.275218 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22"] Nov 02 17:56:24.379328 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.379287 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf8km\" (UniqueName: \"kubernetes.io/projected/d8058f31-67f2-4f71-8f17-5f839c042a2f-kube-api-access-nf8km\") pod \"maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22\" (UID: \"d8058f31-67f2-4f71-8f17-5f839c042a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" Nov 02 17:56:24.379791 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.379452 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8058f31-67f2-4f71-8f17-5f839c042a2f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22\" (UID: \"d8058f31-67f2-4f71-8f17-5f839c042a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" Nov 02 17:56:24.379791 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.379583 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8058f31-67f2-4f71-8f17-5f839c042a2f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22\" (UID: \"d8058f31-67f2-4f71-8f17-5f839c042a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" Nov 02 17:56:24.379791 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.379684 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8058f31-67f2-4f71-8f17-5f839c042a2f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22\" (UID: \"d8058f31-67f2-4f71-8f17-5f839c042a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" Nov 02 17:56:24.481012 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.480970 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8058f31-67f2-4f71-8f17-5f839c042a2f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22\" (UID: \"d8058f31-67f2-4f71-8f17-5f839c042a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" Nov 02 17:56:24.481212 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.481045 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8058f31-67f2-4f71-8f17-5f839c042a2f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22\" (UID: \"d8058f31-67f2-4f71-8f17-5f839c042a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" Nov 02 17:56:24.481340 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.481320 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf8km\" (UniqueName: \"kubernetes.io/projected/d8058f31-67f2-4f71-8f17-5f839c042a2f-kube-api-access-nf8km\") pod \"maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22\" (UID: \"d8058f31-67f2-4f71-8f17-5f839c042a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" Nov 02 17:56:24.481466 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.481447 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8058f31-67f2-4f71-8f17-5f839c042a2f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22\" (UID: \"d8058f31-67f2-4f71-8f17-5f839c042a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" Nov 02 17:56:24.502688 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.486043 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8058f31-67f2-4f71-8f17-5f839c042a2f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22\" (UID: \"d8058f31-67f2-4f71-8f17-5f839c042a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" Nov 02 17:56:24.502942 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.494891 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8058f31-67f2-4f71-8f17-5f839c042a2f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22\" (UID: \"d8058f31-67f2-4f71-8f17-5f839c042a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" Nov 02 17:56:24.507450 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.507397 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf8km\" (UniqueName: \"kubernetes.io/projected/d8058f31-67f2-4f71-8f17-5f839c042a2f-kube-api-access-nf8km\") pod \"maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22\" (UID: \"d8058f31-67f2-4f71-8f17-5f839c042a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" Nov 02 17:56:24.534693 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.529201 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8058f31-67f2-4f71-8f17-5f839c042a2f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22\" (UID: \"d8058f31-67f2-4f71-8f17-5f839c042a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" Nov 02 17:56:24.583817 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.583782 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" Nov 02 17:56:24.857677 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.857550 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" event={"ID":"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a","Type":"ContainerStarted","Data":"93dda336277be2c29190b693bf337279c96299936d5c7ffcb387bba991a6455e"} Nov 02 17:56:24.857677 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.857661 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" event={"ID":"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a","Type":"ContainerStarted","Data":"66909fbf362cc2b6764667d012e84a9dbd483231d7e9a179c10cc75c89bd9f34"} Nov 02 17:56:24.887363 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.887305 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" podStartSLOduration=1.8872900270000001 podStartE2EDuration="1.887290027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:24.882861899 +0000 UTC m=+1349.573915934" watchObservedRunningTime="2026-11-02 17:56:24.887290027 +0000 UTC m=+1349.578344039" Nov 02 17:56:24.891045 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:24.891009 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22"] Nov 02 17:56:24.893044 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:56:24.893015 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8058f31_67f2_4f71_8f17_5f839c042a2f.slice/crio-98b82c902f48315348bc41b2c26046d4e0ac3083a6957d1f722e4ce4a4f461f0 WatchSource:0}: Error finding container 98b82c902f48315348bc41b2c26046d4e0ac3083a6957d1f722e4ce4a4f461f0: Status 404 returned error can't find the container with id 98b82c902f48315348bc41b2c26046d4e0ac3083a6957d1f722e4ce4a4f461f0 Nov 02 17:56:25.477079 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.477042 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:25.483978 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.483948 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:25.829979 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.829951 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" Nov 02 17:56:25.857603 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.857571 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5069cc-7c3d-4f35-8a37-c874e0a834f3-kube-api-access-r24n2\" (UniqueName: \"kubernetes.io/projected/5e5069cc-7c3d-4f35-8a37-c874e0a834f3-kube-api-access-r24n2\") pod \"5e5069cc-7c3d-4f35-8a37-c874e0a834f3\" (UID: \"5e5069cc-7c3d-4f35-8a37-c874e0a834f3\") " Nov 02 17:56:25.857800 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.857697 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5069cc-7c3d-4f35-8a37-c874e0a834f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5069cc-7c3d-4f35-8a37-c874e0a834f3-plugins-config-volume\") pod \"5e5069cc-7c3d-4f35-8a37-c874e0a834f3\" (UID: \"5e5069cc-7c3d-4f35-8a37-c874e0a834f3\") " Nov 02 17:56:25.858248 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.858219 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5069cc-7c3d-4f35-8a37-c874e0a834f3-plugins-config-volume" pod "5e5069cc-7c3d-4f35-8a37-c874e0a834f3" (UID: "5e5069cc-7c3d-4f35-8a37-c874e0a834f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:25.858377 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.858359 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5069cc-7c3d-4f35-8a37-c874e0a834f3-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:25.859919 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.859892 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5069cc-7c3d-4f35-8a37-c874e0a834f3-kube-api-access-r24n2" pod "5e5069cc-7c3d-4f35-8a37-c874e0a834f3" (UID: "5e5069cc-7c3d-4f35-8a37-c874e0a834f3"). InnerVolumeSpecName "kube-api-access-r24n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:25.866738 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.866707 5601 generic.go:356] "Generic (PLEG): container finished" podID="5e5069cc-7c3d-4f35-8a37-c874e0a834f3" containerID="179b56e8bdfd3aa8d9a28ae10b59d097d2c7bd68dad8a1aeaa7b7ddac8a4ec4c" exitCode=0 Nov 02 17:56:25.866873 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.866756 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" Nov 02 17:56:25.866873 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.866792 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" event={"ID":"5e5069cc-7c3d-4f35-8a37-c874e0a834f3","Type":"ContainerDied","Data":"179b56e8bdfd3aa8d9a28ae10b59d097d2c7bd68dad8a1aeaa7b7ddac8a4ec4c"} Nov 02 17:56:25.866873 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.866845 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd" event={"ID":"5e5069cc-7c3d-4f35-8a37-c874e0a834f3","Type":"ContainerDied","Data":"fa20aae5a31da66cc6c5731458ff657eba1968b1e7bbf8dbf68bb17355d0771d"} Nov 02 17:56:25.867031 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.866876 5601 scope.go:122] "RemoveContainer" containerID="179b56e8bdfd3aa8d9a28ae10b59d097d2c7bd68dad8a1aeaa7b7ddac8a4ec4c" Nov 02 17:56:25.869921 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.869238 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" event={"ID":"d8058f31-67f2-4f71-8f17-5f839c042a2f","Type":"ContainerStarted","Data":"cd3ed2b7a2cb6d34a01d609617839afb6a08ef7f4edb83d457ddc6f6c93f734b"} Nov 02 17:56:25.869921 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.869544 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:25.869921 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.869583 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" event={"ID":"d8058f31-67f2-4f71-8f17-5f839c042a2f","Type":"ContainerStarted","Data":"98b82c902f48315348bc41b2c26046d4e0ac3083a6957d1f722e4ce4a4f461f0"} Nov 02 17:56:25.869921 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.869646 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" Nov 02 17:56:25.870923 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.870899 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:56:25.879284 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.879242 5601 scope.go:122] "RemoveContainer" containerID="179b56e8bdfd3aa8d9a28ae10b59d097d2c7bd68dad8a1aeaa7b7ddac8a4ec4c" Nov 02 17:56:25.879635 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:56:25.879598 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"179b56e8bdfd3aa8d9a28ae10b59d097d2c7bd68dad8a1aeaa7b7ddac8a4ec4c\": container with ID starting with 179b56e8bdfd3aa8d9a28ae10b59d097d2c7bd68dad8a1aeaa7b7ddac8a4ec4c not found: ID does not exist" containerID="179b56e8bdfd3aa8d9a28ae10b59d097d2c7bd68dad8a1aeaa7b7ddac8a4ec4c" Nov 02 17:56:25.879745 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.879646 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179b56e8bdfd3aa8d9a28ae10b59d097d2c7bd68dad8a1aeaa7b7ddac8a4ec4c"} err="failed to get container status \"179b56e8bdfd3aa8d9a28ae10b59d097d2c7bd68dad8a1aeaa7b7ddac8a4ec4c\": rpc error: code = NotFound desc = could not find container \"179b56e8bdfd3aa8d9a28ae10b59d097d2c7bd68dad8a1aeaa7b7ddac8a4ec4c\": container with ID starting with 179b56e8bdfd3aa8d9a28ae10b59d097d2c7bd68dad8a1aeaa7b7ddac8a4ec4c not found: ID does not exist" Nov 02 17:56:25.932483 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.932386 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" podStartSLOduration=1.4920203029999999 podStartE2EDuration="1.932365961s" totalImagesPullingTime="440.345658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:25.931442848 +0000 UTC m=+1350.622496861" watchObservedRunningTime="2026-11-02 17:56:25.932365961 +0000 UTC m=+1350.623419975" Nov 02 17:56:25.954603 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.954551 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd"] Nov 02 17:56:25.957732 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.957700 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d8e217-7bfb76d699-sn9cd"] Nov 02 17:56:25.959461 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:25.959430 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r24n2\" (UniqueName: \"kubernetes.io/projected/5e5069cc-7c3d-4f35-8a37-c874e0a834f3-kube-api-access-r24n2\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:27.458655 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:27.457712 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5069cc-7c3d-4f35-8a37-c874e0a834f3" path="/var/lib/kubelet/pods/5e5069cc-7c3d-4f35-8a37-c874e0a834f3/volumes" Nov 02 17:56:31.893018 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:31.892984 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" Nov 02 17:56:33.906547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:33.906501 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p"] Nov 02 17:56:33.907321 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:33.907298 5601 cpu_manager.go:414] "containerMap: removing container" podUID="5e5069cc-7c3d-4f35-8a37-c874e0a834f3" containerName="payload-pre-processing" Nov 02 17:56:33.907419 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:33.907322 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5069cc-7c3d-4f35-8a37-c874e0a834f3" containerName="payload-pre-processing" Nov 02 17:56:33.907510 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:33.907497 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5069cc-7c3d-4f35-8a37-c874e0a834f3" containerName="payload-pre-processing" Nov 02 17:56:33.912340 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:33.912319 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" Nov 02 17:56:33.918849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:33.918826 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-b1a788-serving-cert\"" Nov 02 17:56:33.922973 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:33.922946 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p"] Nov 02 17:56:33.980317 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:33.980269 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc3edc9-ba31-479b-97f7-5eee71b5301b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p\" (UID: \"abc3edc9-ba31-479b-97f7-5eee71b5301b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" Nov 02 17:56:33.980536 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:33.980361 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvkfz\" (UniqueName: \"kubernetes.io/projected/abc3edc9-ba31-479b-97f7-5eee71b5301b-kube-api-access-gvkfz\") pod \"maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p\" (UID: \"abc3edc9-ba31-479b-97f7-5eee71b5301b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" Nov 02 17:56:33.980536 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:33.980480 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abc3edc9-ba31-479b-97f7-5eee71b5301b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p\" (UID: \"abc3edc9-ba31-479b-97f7-5eee71b5301b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" Nov 02 17:56:33.980723 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:33.980566 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abc3edc9-ba31-479b-97f7-5eee71b5301b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p\" (UID: \"abc3edc9-ba31-479b-97f7-5eee71b5301b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" Nov 02 17:56:34.081585 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:34.081546 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abc3edc9-ba31-479b-97f7-5eee71b5301b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p\" (UID: \"abc3edc9-ba31-479b-97f7-5eee71b5301b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" Nov 02 17:56:34.081821 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:34.081705 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc3edc9-ba31-479b-97f7-5eee71b5301b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p\" (UID: \"abc3edc9-ba31-479b-97f7-5eee71b5301b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" Nov 02 17:56:34.081821 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:34.081745 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvkfz\" (UniqueName: \"kubernetes.io/projected/abc3edc9-ba31-479b-97f7-5eee71b5301b-kube-api-access-gvkfz\") pod \"maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p\" (UID: \"abc3edc9-ba31-479b-97f7-5eee71b5301b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" Nov 02 17:56:34.081945 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:34.081839 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abc3edc9-ba31-479b-97f7-5eee71b5301b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p\" (UID: \"abc3edc9-ba31-479b-97f7-5eee71b5301b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" Nov 02 17:56:34.082393 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:34.082365 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc3edc9-ba31-479b-97f7-5eee71b5301b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p\" (UID: \"abc3edc9-ba31-479b-97f7-5eee71b5301b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" Nov 02 17:56:34.084178 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:34.084155 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abc3edc9-ba31-479b-97f7-5eee71b5301b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p\" (UID: \"abc3edc9-ba31-479b-97f7-5eee71b5301b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" Nov 02 17:56:34.084280 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:34.084219 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abc3edc9-ba31-479b-97f7-5eee71b5301b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p\" (UID: \"abc3edc9-ba31-479b-97f7-5eee71b5301b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" Nov 02 17:56:34.093753 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:34.093722 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvkfz\" (UniqueName: \"kubernetes.io/projected/abc3edc9-ba31-479b-97f7-5eee71b5301b-kube-api-access-gvkfz\") pod \"maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p\" (UID: \"abc3edc9-ba31-479b-97f7-5eee71b5301b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" Nov 02 17:56:34.227928 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:34.227892 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" Nov 02 17:56:34.538400 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:34.538348 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p"] Nov 02 17:56:34.556761 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:56:34.556697 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:56:34.953436 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:34.953409 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" event={"ID":"abc3edc9-ba31-479b-97f7-5eee71b5301b","Type":"ContainerStarted","Data":"8193a8acee95e0d75121d240380edc178d26af838b309f02189df160895a998c"} Nov 02 17:56:35.974004 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:35.973965 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" event={"ID":"abc3edc9-ba31-479b-97f7-5eee71b5301b","Type":"ContainerStarted","Data":"7a50bde6181623e4669f49babdb8a29d88b3133266fa0b2f92099f067188c88a"} Nov 02 17:56:35.975651 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:35.975598 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" Nov 02 17:56:36.015562 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:36.015493 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" podStartSLOduration=2.625428534 podStartE2EDuration="3.01547121s" totalImagesPullingTime="390.042676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:35.997459183 +0000 UTC m=+1360.688513195" watchObservedRunningTime="2026-11-02 17:56:36.01547121 +0000 UTC m=+1360.706525223" Nov 02 17:56:43.010285 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:56:43.010203 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" Nov 02 17:57:04.486722 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:57:04.486679 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:57:08.713791 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.713750 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb"] Nov 02 17:57:08.719832 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.719793 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.735362 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.733827 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-b6b6f4-openshift-default-dockercfg-rnffh\"" Nov 02 17:57:08.745865 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.745824 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb"] Nov 02 17:57:08.791971 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.790952 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.791971 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.791033 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.791971 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.791082 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m62h\" (UniqueName: \"kubernetes.io/projected/62dd825e-e164-4e44-b5ec-4c003f89c113-kube-api-access-7m62h\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.791971 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.791161 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.791971 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.791284 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.791971 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.791383 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dd825e-e164-4e44-b5ec-4c003f89c113-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.791971 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.791483 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.791971 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.791532 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.794018 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.793983 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.895536 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.895495 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.895741 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.895601 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.895741 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.895682 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.895874 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.895748 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.895874 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.895803 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m62h\" (UniqueName: \"kubernetes.io/projected/62dd825e-e164-4e44-b5ec-4c003f89c113-kube-api-access-7m62h\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.895874 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.895841 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.896027 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.895890 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.896027 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.895974 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.896027 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.895989 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dd825e-e164-4e44-b5ec-4c003f89c113-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.896185 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.896140 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.896336 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.896312 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.896628 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.896472 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.896802 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.896694 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.896802 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.896760 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dd825e-e164-4e44-b5ec-4c003f89c113-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.898083 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.898054 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.898464 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.898444 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.936276 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.936242 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m62h\" (UniqueName: \"kubernetes.io/projected/62dd825e-e164-4e44-b5ec-4c003f89c113-kube-api-access-7m62h\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:08.959327 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:08.959233 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:09.044117 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:09.044015 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:09.357584 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:09.357487 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb"] Nov 02 17:57:09.361239 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:57:09.361211 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62dd825e_e164_4e44_b5ec_4c003f89c113.slice/crio-c2ef3958e272c7823401fbb9837210558a036cda029cc78f028586519f13e3b9 WatchSource:0}: Error finding container c2ef3958e272c7823401fbb9837210558a036cda029cc78f028586519f13e3b9: Status 404 returned error can't find the container with id c2ef3958e272c7823401fbb9837210558a036cda029cc78f028586519f13e3b9 Nov 02 17:57:09.364271 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:09.364236 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:09.364369 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:09.364309 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:09.364369 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:09.364339 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:10.368651 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:10.368313 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" event={"ID":"62dd825e-e164-4e44-b5ec-4c003f89c113","Type":"ContainerStarted","Data":"2db54e9965a17344a6bb31b5f040421faa2a57c2fd457b420b585341937dd990"} Nov 02 17:57:10.368651 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:10.368411 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" event={"ID":"62dd825e-e164-4e44-b5ec-4c003f89c113","Type":"ContainerStarted","Data":"c2ef3958e272c7823401fbb9837210558a036cda029cc78f028586519f13e3b9"} Nov 02 17:57:10.402411 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:10.402348 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" podStartSLOduration=2.402329391 podStartE2EDuration="2.402329391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:10.399607781 +0000 UTC m=+1395.090661790" watchObservedRunningTime="2026-11-02 17:57:10.402329391 +0000 UTC m=+1395.093383402" Nov 02 17:57:11.052638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:11.047844 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:11.061642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:11.059760 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 17:57:11.061642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:11.059846 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" podUID="62dd825e-e164-4e44-b5ec-4c003f89c113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 17:57:12.047641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:12.045020 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 17:57:12.047641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:12.045097 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" podUID="62dd825e-e164-4e44-b5ec-4c003f89c113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 17:57:13.051662 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:13.046221 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 17:57:13.051662 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:13.046309 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" podUID="62dd825e-e164-4e44-b5ec-4c003f89c113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 17:57:14.046642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:14.046138 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 17:57:14.046642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:14.046220 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" podUID="62dd825e-e164-4e44-b5ec-4c003f89c113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 17:57:15.049642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:15.048055 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 17:57:15.049642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:15.048120 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" podUID="62dd825e-e164-4e44-b5ec-4c003f89c113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 17:57:15.916492 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:15.916459 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m"] Nov 02 17:57:15.920865 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:15.920836 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:15.925472 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:15.925448 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-1e2281-openshift-default-dockercfg-xhl4q\"" Nov 02 17:57:15.937754 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:15.937703 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m"] Nov 02 17:57:16.007566 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.007525 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.007738 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.007671 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.007804 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.007782 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.007858 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.007844 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.007992 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.007974 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25416f1c-51e9-48fe-a6c7-4382172adaf5-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.008054 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.008005 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29kxn\" (UniqueName: \"kubernetes.io/projected/25416f1c-51e9-48fe-a6c7-4382172adaf5-kube-api-access-29kxn\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.008092 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.008051 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.008092 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.008079 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.008179 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.008121 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.049104 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.049072 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:16.049586 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.049569 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:16.050244 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.050227 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 17:57:16.110302 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.110263 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.110484 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.110376 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.110484 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.110420 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.110646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.110589 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25416f1c-51e9-48fe-a6c7-4382172adaf5-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.110646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.110611 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29kxn\" (UniqueName: \"kubernetes.io/projected/25416f1c-51e9-48fe-a6c7-4382172adaf5-kube-api-access-29kxn\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.110757 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.110676 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.110757 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.110718 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.110847 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.110800 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.110847 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.110834 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.111674 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.111199 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.111674 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.111548 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.111674 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.111658 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.111674 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.111669 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25416f1c-51e9-48fe-a6c7-4382172adaf5-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.112566 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.112542 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.113533 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.113509 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.114547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.114525 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.141582 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.140634 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29kxn\" (UniqueName: \"kubernetes.io/projected/25416f1c-51e9-48fe-a6c7-4382172adaf5-kube-api-access-29kxn\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.145205 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.145176 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.219869 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.219838 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw"] Nov 02 17:57:16.225319 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.225295 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.236925 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.236896 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-b1a788-openshift-default-dockercfg-kv6gv\"" Nov 02 17:57:16.237083 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.237058 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw"] Nov 02 17:57:16.266327 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.266294 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:16.314195 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.314156 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-data\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.314367 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.314218 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-workload-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.314424 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.314346 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-token\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.314477 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.314445 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-workload-certs\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.314529 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.314483 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-envoy\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.314589 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.314535 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-credential-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.314673 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.314605 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mffsw\" (UniqueName: \"kubernetes.io/projected/ee0d3ab7-7076-4998-b997-2fa88e7622ad-kube-api-access-mffsw\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.314780 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.314767 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istiod-ca-cert\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.314838 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.314798 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-podinfo\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.416724 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.416681 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-token\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.416883 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.416790 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-workload-certs\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.416883 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.416818 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-envoy\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.416964 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.416902 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-credential-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.416964 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.416927 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mffsw\" (UniqueName: \"kubernetes.io/projected/ee0d3ab7-7076-4998-b997-2fa88e7622ad-kube-api-access-mffsw\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.417039 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.417030 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istiod-ca-cert\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.417111 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.417096 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-podinfo\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.417161 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.417150 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-data\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.417200 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.417192 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-workload-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.417537 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.417521 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-workload-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.417986 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.417966 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-workload-certs\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.418988 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.418963 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istiod-ca-cert\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.419225 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.419202 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-credential-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.419742 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.419718 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-data\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.421228 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.421202 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-podinfo\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.425279 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.422506 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-envoy\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.435327 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.433695 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-token\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.438301 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.438275 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mffsw\" (UniqueName: \"kubernetes.io/projected/ee0d3ab7-7076-4998-b997-2fa88e7622ad-kube-api-access-mffsw\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.545671 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.545561 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:16.580770 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.580746 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m"] Nov 02 17:57:16.581425 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:57:16.581397 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25416f1c_51e9_48fe_a6c7_4382172adaf5.slice/crio-8692c43d95e523e96abeb7d55a163ca7e0acb9a7f88a13a16c1d083a29ee9293 WatchSource:0}: Error finding container 8692c43d95e523e96abeb7d55a163ca7e0acb9a7f88a13a16c1d083a29ee9293: Status 404 returned error can't find the container with id 8692c43d95e523e96abeb7d55a163ca7e0acb9a7f88a13a16c1d083a29ee9293 Nov 02 17:57:16.584234 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.584199 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:16.584308 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.584268 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:16.584308 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.584298 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:16.838138 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.838097 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw"] Nov 02 17:57:16.838580 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:57:16.838555 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee0d3ab7_7076_4998_b997_2fa88e7622ad.slice/crio-0dd85333ed1b093c5cdf1d4ad4b1467011b41fffcd13690ee2c7c8c1a1ad07a4 WatchSource:0}: Error finding container 0dd85333ed1b093c5cdf1d4ad4b1467011b41fffcd13690ee2c7c8c1a1ad07a4: Status 404 returned error can't find the container with id 0dd85333ed1b093c5cdf1d4ad4b1467011b41fffcd13690ee2c7c8c1a1ad07a4 Nov 02 17:57:16.841606 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.841567 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:16.841750 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.841669 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:16.841750 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:16.841714 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:17.441457 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:17.441421 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" event={"ID":"ee0d3ab7-7076-4998-b997-2fa88e7622ad","Type":"ContainerStarted","Data":"4cdec65986f89463578aabf9ff0a1b1f818b048b74925193184dc1a3f97eef06"} Nov 02 17:57:17.441966 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:17.441483 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" event={"ID":"ee0d3ab7-7076-4998-b997-2fa88e7622ad","Type":"ContainerStarted","Data":"0dd85333ed1b093c5cdf1d4ad4b1467011b41fffcd13690ee2c7c8c1a1ad07a4"} Nov 02 17:57:17.443978 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:17.443952 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" event={"ID":"25416f1c-51e9-48fe-a6c7-4382172adaf5","Type":"ContainerStarted","Data":"b2e66dae7643aad11ab2cd6720cc4b6f43e5df78874a1bee1ff1ffc8bfa7da36"} Nov 02 17:57:17.444101 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:17.444002 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" event={"ID":"25416f1c-51e9-48fe-a6c7-4382172adaf5","Type":"ContainerStarted","Data":"8692c43d95e523e96abeb7d55a163ca7e0acb9a7f88a13a16c1d083a29ee9293"} Nov 02 17:57:17.484455 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:17.484383 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" podStartSLOduration=1.484364275 podStartE2EDuration="1.484364275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:17.479526464 +0000 UTC m=+1402.170580501" watchObservedRunningTime="2026-11-02 17:57:17.484364275 +0000 UTC m=+1402.175418288" Nov 02 17:57:17.513224 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:17.513159 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" podStartSLOduration=2.513140917 podStartE2EDuration="2.513140917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:17.5100129 +0000 UTC m=+1402.201066912" watchObservedRunningTime="2026-11-02 17:57:17.513140917 +0000 UTC m=+1402.204194931" Nov 02 17:57:17.551664 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:17.549008 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:17.551664 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:17.549139 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 17:57:17.551664 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:17.549180 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" podUID="ee0d3ab7-7076-4998-b997-2fa88e7622ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 17:57:18.273162 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:18.270024 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:18.288639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:18.278758 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:57:18.288639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:18.278825 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" podUID="25416f1c-51e9-48fe-a6c7-4382172adaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 02 17:57:18.549164 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:18.549061 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 17:57:18.549863 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:18.549820 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" podUID="ee0d3ab7-7076-4998-b997-2fa88e7622ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 17:57:19.268640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:19.268064 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:57:19.268640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:19.268132 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" podUID="25416f1c-51e9-48fe-a6c7-4382172adaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 02 17:57:19.554693 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:19.546000 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 17:57:19.554693 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:19.546066 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" podUID="ee0d3ab7-7076-4998-b997-2fa88e7622ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 17:57:20.270642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:20.270026 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:57:20.270642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:20.270089 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" podUID="25416f1c-51e9-48fe-a6c7-4382172adaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 02 17:57:20.557697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:20.557176 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 17:57:20.557697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:20.557248 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" podUID="ee0d3ab7-7076-4998-b997-2fa88e7622ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 17:57:21.271710 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:21.267170 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:57:21.271710 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:21.267262 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" podUID="25416f1c-51e9-48fe-a6c7-4382172adaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 02 17:57:21.580701 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:21.564337 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 17:57:21.580701 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:21.564408 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" podUID="ee0d3ab7-7076-4998-b997-2fa88e7622ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 17:57:22.267063 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:22.267024 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:57:22.267374 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:22.267343 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" podUID="25416f1c-51e9-48fe-a6c7-4382172adaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 02 17:57:22.555691 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:22.552017 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 17:57:22.555691 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:22.552081 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" podUID="ee0d3ab7-7076-4998-b997-2fa88e7622ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 17:57:23.003691 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.003654 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm"] Nov 02 17:57:23.021006 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.020962 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm"] Nov 02 17:57:23.021965 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.021924 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.031966 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.031908 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-9165c3-openshift-default-dockercfg-7wchl\"" Nov 02 17:57:23.158640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.149607 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pjdn\" (UniqueName: \"kubernetes.io/projected/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-kube-api-access-8pjdn\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.158640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.149744 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.158640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.149801 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.158640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.149850 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.158640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.149896 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.158640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.149937 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.158640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.150069 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.158640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.150116 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.158640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.150275 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.251996 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.251947 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.252179 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.252151 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.252229 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.252211 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pjdn\" (UniqueName: \"kubernetes.io/projected/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-kube-api-access-8pjdn\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.252269 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.252257 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.252308 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.252297 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.252366 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.252351 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.252418 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.252400 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.252471 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.252442 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.252873 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.252436 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.252873 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.252661 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.252873 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.252826 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.253103 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.253043 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.253164 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.253112 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.253446 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.253419 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.255108 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.255039 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.255482 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.255460 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.265194 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.265174 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.265429 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.265407 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pjdn\" (UniqueName: \"kubernetes.io/projected/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-kube-api-access-8pjdn\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.267242 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.267219 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:57:23.267346 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.267270 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" podUID="25416f1c-51e9-48fe-a6c7-4382172adaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 02 17:57:23.418867 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.418655 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:23.678876 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.678835 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:23.680815 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.680788 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:23.685901 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.685871 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 17:57:23.788666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.780921 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r"] Nov 02 17:57:23.788666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.787158 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:23.937011 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.936970 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:23.937260 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.937240 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:23.937397 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.937379 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:23.937559 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.937545 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc1c481d-b715-4ffd-8803-7371e21eead0-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:23.937730 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.937712 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgbls\" (UniqueName: \"kubernetes.io/projected/dc1c481d-b715-4ffd-8803-7371e21eead0-kube-api-access-xgbls\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:23.937872 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.937857 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:23.938052 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.938038 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:23.938176 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.938163 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:23.938304 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.938290 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:23.977803 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:23.977690 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r"] Nov 02 17:57:24.028905 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.028510 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm"] Nov 02 17:57:24.035112 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.035082 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.042646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.042550 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgbls\" (UniqueName: \"kubernetes.io/projected/dc1c481d-b715-4ffd-8803-7371e21eead0-kube-api-access-xgbls\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:24.042646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.042644 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:24.042892 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.042770 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:24.042892 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.042821 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:24.042892 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.042880 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:24.043089 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.042952 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:24.043089 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.042976 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:24.043089 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.043035 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:24.043245 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.043126 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc1c481d-b715-4ffd-8803-7371e21eead0-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:24.046636 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.044292 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc1c481d-b715-4ffd-8803-7371e21eead0-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:24.050925 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.049008 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm"] Nov 02 17:57:24.050925 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.049607 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:24.050925 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.050688 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:24.051159 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.051015 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:24.053874 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.051267 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:24.060499 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.060448 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:24.065836 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.063465 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgbls\" (UniqueName: \"kubernetes.io/projected/dc1c481d-b715-4ffd-8803-7371e21eead0-kube-api-access-xgbls\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:24.065836 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.064631 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:24.070349 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.068195 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:24.070349 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.068345 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:24.070349 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.068574 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:24.097276 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.085320 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm"] Nov 02 17:57:24.105827 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.105762 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:24.145751 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.145707 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.146087 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.146063 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvl8x\" (UniqueName: \"kubernetes.io/projected/64eaca4e-6457-4551-8e3d-6188dc62b363-kube-api-access-pvl8x\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.146419 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.146395 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.146639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.146599 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.146777 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.146760 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64eaca4e-6457-4551-8e3d-6188dc62b363-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.146914 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.146898 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.147105 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.147088 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.147301 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.147284 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.147540 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.147522 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.249123 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.249084 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvl8x\" (UniqueName: \"kubernetes.io/projected/64eaca4e-6457-4551-8e3d-6188dc62b363-kube-api-access-pvl8x\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.249467 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.249440 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.250887 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.250853 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.260796 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.260749 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.260985 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.260831 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64eaca4e-6457-4551-8e3d-6188dc62b363-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.260985 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.260906 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.261248 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.261218 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.261412 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.261386 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.261412 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.261407 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.261764 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.261732 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.261940 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.261918 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.262115 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.262095 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.263825 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.262950 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.269415 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.269392 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64eaca4e-6457-4551-8e3d-6188dc62b363-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.279793 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.275067 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.282425 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.282372 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.283462 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.283382 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvl8x\" (UniqueName: \"kubernetes.io/projected/64eaca4e-6457-4551-8e3d-6188dc62b363-kube-api-access-pvl8x\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.284575 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.284551 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.319903 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.318496 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:24.320686 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.320664 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:24.321871 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.321851 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 17:57:24.388944 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.388905 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:24.470478 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.468968 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:24.678389 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.677243 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" event={"ID":"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb","Type":"ContainerStarted","Data":"ef9e829b4a0b0daafbf84838306e4bc44c37cde7a2f04a06de8872a552f5de88"} Nov 02 17:57:24.678389 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.677301 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" event={"ID":"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb","Type":"ContainerStarted","Data":"12138ded2aae95778cde2b27e8d22bae69709ceb9ad514ca7cfdd33a258a37c7"} Nov 02 17:57:24.712419 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.710020 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" podStartSLOduration=2.710002639 podStartE2EDuration="2.710002639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:24.7093278 +0000 UTC m=+1409.400381813" watchObservedRunningTime="2026-11-02 17:57:24.710002639 +0000 UTC m=+1409.401056651" Nov 02 17:57:24.967760 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.964528 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r"] Nov 02 17:57:24.979327 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:57:24.979297 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1c481d_b715_4ffd_8803_7371e21eead0.slice/crio-37094bd7fcf2b0dd6dbe3b06792e79ced12ce078441c8b307468056b089e16f1 WatchSource:0}: Error finding container 37094bd7fcf2b0dd6dbe3b06792e79ced12ce078441c8b307468056b089e16f1: Status 404 returned error can't find the container with id 37094bd7fcf2b0dd6dbe3b06792e79ced12ce078441c8b307468056b089e16f1 Nov 02 17:57:24.987204 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.987129 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:24.987514 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.987473 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:24.987726 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:24.987689 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:25.118297 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:25.118266 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm"] Nov 02 17:57:25.120406 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:57:25.120379 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64eaca4e_6457_4551_8e3d_6188dc62b363.slice/crio-a8735216eb50873b97e2575f2a4d03baaa91a3f15425d433cd95e99c54179d87 WatchSource:0}: Error finding container a8735216eb50873b97e2575f2a4d03baaa91a3f15425d433cd95e99c54179d87: Status 404 returned error can't find the container with id a8735216eb50873b97e2575f2a4d03baaa91a3f15425d433cd95e99c54179d87 Nov 02 17:57:25.124002 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:25.123959 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:25.124086 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:25.124058 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:25.124129 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:25.124112 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:25.444641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:25.439424 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:25.444641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:25.440715 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 17:57:25.444641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:25.440771 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" podUID="29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 17:57:25.713480 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:25.713347 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" event={"ID":"64eaca4e-6457-4551-8e3d-6188dc62b363","Type":"ContainerStarted","Data":"8e7bdc2ddd8fc45acc19641a5a4867ed9c2c85c81cb5cd4c8814fa3d8bc5b15f"} Nov 02 17:57:25.713850 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:25.713822 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" event={"ID":"64eaca4e-6457-4551-8e3d-6188dc62b363","Type":"ContainerStarted","Data":"a8735216eb50873b97e2575f2a4d03baaa91a3f15425d433cd95e99c54179d87"} Nov 02 17:57:25.727555 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:25.727251 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" event={"ID":"dc1c481d-b715-4ffd-8803-7371e21eead0","Type":"ContainerStarted","Data":"ccbc8fe3bf80234c11ddfbd335950e0c8faaf9b9e845352f1514aa5bc936428a"} Nov 02 17:57:25.727555 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:25.727294 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" event={"ID":"dc1c481d-b715-4ffd-8803-7371e21eead0","Type":"ContainerStarted","Data":"37094bd7fcf2b0dd6dbe3b06792e79ced12ce078441c8b307468056b089e16f1"} Nov 02 17:57:25.793818 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:25.793751 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" podStartSLOduration=2.793730942 podStartE2EDuration="2.793730942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:25.754392343 +0000 UTC m=+1410.445446357" watchObservedRunningTime="2026-11-02 17:57:25.793730942 +0000 UTC m=+1410.484784954" Nov 02 17:57:26.125848 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.120047 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" podStartSLOduration=3.120025403 podStartE2EDuration="3.120025403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:25.792993651 +0000 UTC m=+1410.484047664" watchObservedRunningTime="2026-11-02 17:57:26.120025403 +0000 UTC m=+1410.811079416" Nov 02 17:57:26.228639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.227812 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7"] Nov 02 17:57:26.261640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.260815 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" Nov 02 17:57:26.291454 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.291423 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-23a657-serving-cert\"" Nov 02 17:57:26.310266 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.307589 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7"] Nov 02 17:57:26.604739 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.493458 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:26.604739 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.528786 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 17:57:26.604739 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.528857 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" podUID="29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 17:57:26.604739 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.572246 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:26.604739 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.572389 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:57:26.604739 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.572446 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" podUID="64eaca4e-6457-4551-8e3d-6188dc62b363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 17:57:26.625717 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.605458 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:57:26.625717 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.605523 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" podUID="dc1c481d-b715-4ffd-8803-7371e21eead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 17:57:26.649659 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.642739 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-maas-api-tls\") pod \"maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7\" (UID: \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" Nov 02 17:57:26.649659 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.642805 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-openshift-service-ca\") pod \"maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7\" (UID: \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" Nov 02 17:57:26.649659 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.642862 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d46kb\" (UniqueName: \"kubernetes.io/projected/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-kube-api-access-d46kb\") pod \"maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7\" (UID: \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" Nov 02 17:57:26.649659 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.643104 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-metrics-certs\") pod \"maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7\" (UID: \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" Nov 02 17:57:26.770644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.757366 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-metrics-certs\") pod \"maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7\" (UID: \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" Nov 02 17:57:26.771248 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.771211 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-maas-api-tls\") pod \"maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7\" (UID: \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" Nov 02 17:57:26.771466 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.771439 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-openshift-service-ca\") pod \"maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7\" (UID: \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" Nov 02 17:57:26.771675 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.771655 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d46kb\" (UniqueName: \"kubernetes.io/projected/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-kube-api-access-d46kb\") pod \"maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7\" (UID: \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" Nov 02 17:57:26.805598 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:57:26.773180 5601 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-serving-cert: secret "maas-api-e2e-explicit-23a657-serving-cert" not found Nov 02 17:57:26.805930 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:57:26.805906 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-maas-api-tls podName:64fe7612-3ce2-451e-bcd1-0f80ebd1f352 nodeName:}" failed. No retries permitted until 2026-11-02 17:57:27.305861396 +0000 UTC m=+1411.996915403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-maas-api-tls") pod "maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" (UID: "64fe7612-3ce2-451e-bcd1-0f80ebd1f352") : secret "maas-api-e2e-explicit-23a657-serving-cert" not found Nov 02 17:57:26.806333 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.789915 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-openshift-service-ca\") pod \"maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7\" (UID: \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" Nov 02 17:57:26.834892 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.834856 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-metrics-certs\") pod \"maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7\" (UID: \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" Nov 02 17:57:26.949646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:26.943390 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d46kb\" (UniqueName: \"kubernetes.io/projected/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-kube-api-access-d46kb\") pod \"maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7\" (UID: \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" Nov 02 17:57:27.323693 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:27.310799 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 02 17:57:27.460734 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:27.460700 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:57:27.461368 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:27.461324 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" podUID="dc1c481d-b715-4ffd-8803-7371e21eead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 17:57:27.497911 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:27.494993 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:57:27.497911 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:27.495048 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" podUID="64eaca4e-6457-4551-8e3d-6188dc62b363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 17:57:27.497911 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:27.496945 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 17:57:27.497911 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:27.497702 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:27.505038 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:27.504251 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-maas-api-tls\") pod \"maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7\" (UID: \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" Nov 02 17:57:27.505321 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:27.505298 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 17:57:27.505458 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:27.505438 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" podUID="29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 17:57:27.531184 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:27.531147 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:27.540988 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:27.540952 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 17:57:27.587342 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:27.587252 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-maas-api-tls\") pod \"maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7\" (UID: \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" Nov 02 17:57:27.785641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:27.784000 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" Nov 02 17:57:28.238637 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:28.238577 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Nov 02 17:57:28.285643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:28.281738 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:28.317641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:28.317293 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:28.337641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:28.333563 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:57:28.444641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:28.411829 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:57:28.444641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:28.411889 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" podUID="dc1c481d-b715-4ffd-8803-7371e21eead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 17:57:28.471637 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:28.446553 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 17:57:28.471637 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:28.446610 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" podUID="29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 17:57:28.519795 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:28.473554 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:57:28.519795 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:28.473609 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" podUID="64eaca4e-6457-4551-8e3d-6188dc62b363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 17:57:28.519795 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:28.502127 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:28.519795 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:28.502469 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 17:57:28.519795 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:28.502796 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:57:28.519795 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:28.502854 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:57:28.546642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:28.539247 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 17:57:28.546642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:28.539649 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:57:28.546642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:28.541399 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 17:57:28.546642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:28.541702 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:57:28.654047 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:28.635763 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:29.325642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:29.322731 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:29.394601 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:29.394177 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:57:29.394601 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:29.394235 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" podUID="dc1c481d-b715-4ffd-8803-7371e21eead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 17:57:29.447440 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:29.447404 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 17:57:29.447769 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:29.447736 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" podUID="29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 17:57:29.492652 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:29.471099 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:57:29.493031 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:29.492985 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" podUID="64eaca4e-6457-4551-8e3d-6188dc62b363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 17:57:29.499649 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:29.489184 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:29.504736 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:29.504697 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:29.505230 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:29.505202 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 02 17:57:29.542746 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:29.542706 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Nov 02 17:57:29.590830 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:29.590736 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:29.623884 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:29.623847 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:29.636883 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:29.636840 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:29.637385 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:29.637345 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:57:29.637495 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:29.637400 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7"] Nov 02 17:57:29.810660 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:29.803736 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 17:57:30.307689 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:30.307019 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:57:30.327750 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:30.327703 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:30.415642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:30.396578 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:57:30.483006 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:30.482965 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" podUID="dc1c481d-b715-4ffd-8803-7371e21eead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 17:57:30.485105 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:30.451746 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 17:57:30.485338 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:30.485280 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" podUID="29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 17:57:30.503354 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:30.503321 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:57:30.503578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:30.503552 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" podUID="64eaca4e-6457-4551-8e3d-6188dc62b363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 17:57:30.563775 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:30.563732 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:30.595964 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:30.595901 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:30.626358 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:30.626315 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:30.664723 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:30.664680 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 17:57:30.920186 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:30.916749 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" event={"ID":"64fe7612-3ce2-451e-bcd1-0f80ebd1f352","Type":"ContainerStarted","Data":"2c0adfc40205136ffc8c89d9c3456c1b8d9368bfda289848cd4f2c46315a15e6"} Nov 02 17:57:31.347639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:31.338992 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 02 17:57:31.420721 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:31.406547 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:57:31.420721 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:31.406630 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" podUID="dc1c481d-b715-4ffd-8803-7371e21eead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 17:57:31.458707 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:31.454792 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 17:57:31.459121 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:31.459080 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" podUID="29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 17:57:31.471816 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:31.471782 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:57:31.472045 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:31.472020 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" podUID="64eaca4e-6457-4551-8e3d-6188dc62b363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 17:57:31.489797 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:31.489754 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 02 17:57:31.512608 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:31.512566 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:31.570097 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:31.570047 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 02 17:57:31.597985 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:31.597881 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:31.629746 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:31.629701 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:31.764304 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:31.763590 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 17:57:31.808360 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:31.808312 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:32.000666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:31.997056 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" event={"ID":"64fe7612-3ce2-451e-bcd1-0f80ebd1f352","Type":"ContainerStarted","Data":"3af6786929a709584db8f48a600250ba78b5a7e597bbd5633ec592d7b59fae07"} Nov 02 17:57:32.000666 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:31.998433 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" Nov 02 17:57:32.032642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.030972 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" podStartSLOduration=5.341596718 podStartE2EDuration="6.030949677s" totalImagesPullingTime="689.352959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:32.020185647 +0000 UTC m=+1416.711239660" watchObservedRunningTime="2026-11-02 17:57:32.030949677 +0000 UTC m=+1416.722003691" Nov 02 17:57:32.128741 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.128694 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:32.498657 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.474590 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 02 17:57:32.501732 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.501538 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 17:57:32.501732 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.501591 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" podUID="29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 17:57:32.515868 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.502437 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:57:32.515868 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.502482 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" podUID="64eaca4e-6457-4551-8e3d-6188dc62b363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 17:57:32.515868 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.503224 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:32.515868 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.504042 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 17:57:32.515868 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.505028 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:32.526695 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.517502 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:57:32.526695 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.521939 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:57:32.526695 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.521991 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" podUID="dc1c481d-b715-4ffd-8803-7371e21eead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 17:57:32.526695 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.522066 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:32.526695 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.522309 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:32.526695 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.522554 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:57:32.582905 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.581752 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:32.631845 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.614759 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:32.662075 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.659874 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 17:57:32.705101 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:32.705050 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:33.132074 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.132047 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 17:57:33.404696 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.404229 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:57:33.404696 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.404308 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" podUID="dc1c481d-b715-4ffd-8803-7371e21eead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 17:57:33.489645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.488589 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:57:33.489645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.488667 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" podUID="64eaca4e-6457-4551-8e3d-6188dc62b363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 17:57:33.529747 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.529288 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:33.531080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.530332 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 02 17:57:33.540741 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.540706 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:33.541370 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.541231 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:57:33.584547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.584491 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 02 17:57:33.638694 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.632121 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 17:57:33.667072 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.666982 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:33.668633 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.668596 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:33.693888 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.693860 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 17:57:33.694099 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.693704 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:33.708258 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.699565 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 17:57:33.718647 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.718191 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:33.805352 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.805309 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 17:57:33.863658 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.863593 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:33.933134 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.933104 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 17:57:33.956032 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:33.955934 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 17:57:34.050684 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:34.048837 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:34.390079 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:34.389988 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:57:34.390324 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:34.390297 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" podUID="dc1c481d-b715-4ffd-8803-7371e21eead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 17:57:34.489015 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:34.488984 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:57:34.489283 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:34.489253 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" podUID="64eaca4e-6457-4551-8e3d-6188dc62b363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 17:57:34.535391 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:34.535350 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:34.536579 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:57:34.536542 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:57:34.654883 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:34.654799 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 17:57:35.432183 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:35.432147 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:35.433905 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:35.433881 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:35.436554 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:35.436534 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 17:57:35.477460 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:35.477428 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:35.477741 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:35.477723 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:35.478952 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:35.478929 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 17:57:39.341046 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:39.340957 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" Nov 02 17:57:39.391227 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:39.391189 5601 generic.go:356] "Generic (PLEG): container finished" podID="0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" containerID="93dda336277be2c29190b693bf337279c96299936d5c7ffcb387bba991a6455e" exitCode=0 Nov 02 17:57:39.391420 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:39.391288 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" event={"ID":"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a","Type":"ContainerDied","Data":"93dda336277be2c29190b693bf337279c96299936d5c7ffcb387bba991a6455e"} Nov 02 17:57:39.391812 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:39.391785 5601 scope.go:122] "RemoveContainer" containerID="93dda336277be2c29190b693bf337279c96299936d5c7ffcb387bba991a6455e" Nov 02 17:57:39.393930 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:39.393892 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:39.394054 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:39.393986 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:39.394054 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:39.394033 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:40.399984 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:40.399944 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" event={"ID":"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a","Type":"ContainerStarted","Data":"84c9171fa2cacf21d0e7db8324e5f6ff5a09e7c2938af1e60bb3b39ddacf01d5"} Nov 02 17:57:40.476677 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:40.476606 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:57:40.477551 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:40.477525 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:57:40.477774 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:40.477746 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" podUID="0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 17:57:40.869598 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:40.869560 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:57:41.477201 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:41.477171 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:57:41.477600 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:41.477220 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" podUID="0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 17:57:42.477193 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:42.477154 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:57:42.477372 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:42.477227 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" podUID="0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 17:57:43.477222 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:43.477188 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:57:43.477419 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:43.477246 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" podUID="0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 17:57:44.477147 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:44.477107 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:57:44.477442 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:44.477410 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" podUID="0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 17:57:45.477199 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:45.477148 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:57:45.477382 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:45.477224 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" podUID="0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 17:57:45.851751 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:45.851659 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj"] Nov 02 17:57:45.855846 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:45.855828 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:45.868650 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:45.868587 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-86f5f0-openshift-default-dockercfg-lwhdl\"" Nov 02 17:57:45.869497 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:45.869468 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj"] Nov 02 17:57:45.905672 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:45.905631 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196bcedf-fdfd-4469-9234-d281f907d53a-istio-token\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:45.905910 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:45.905885 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-credential-socket\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:45.906052 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:45.906028 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stx8m\" (UniqueName: \"kubernetes.io/projected/196bcedf-fdfd-4469-9234-d281f907d53a-kube-api-access-stx8m\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:45.906110 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:45.906080 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196bcedf-fdfd-4469-9234-d281f907d53a-istiod-ca-cert\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:45.906252 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:45.906227 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-istio-envoy\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:45.906375 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:45.906359 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-workload-socket\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:45.906477 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:45.906423 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-workload-certs\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:45.906649 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:45.906606 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-istio-data\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:45.906824 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:45.906806 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196bcedf-fdfd-4469-9234-d281f907d53a-istio-podinfo\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.010638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.009046 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196bcedf-fdfd-4469-9234-d281f907d53a-istio-podinfo\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.010638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.009132 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196bcedf-fdfd-4469-9234-d281f907d53a-istio-token\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.010638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.009177 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-credential-socket\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.010638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.009269 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stx8m\" (UniqueName: \"kubernetes.io/projected/196bcedf-fdfd-4469-9234-d281f907d53a-kube-api-access-stx8m\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.010638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.009303 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196bcedf-fdfd-4469-9234-d281f907d53a-istiod-ca-cert\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.010638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.009343 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-istio-envoy\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.010638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.009456 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-workload-socket\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.010638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.009518 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-workload-certs\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.010638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.009567 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-istio-data\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.010638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.010277 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-istio-data\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.011342 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.011310 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196bcedf-fdfd-4469-9234-d281f907d53a-istiod-ca-cert\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.015457 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.011876 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-credential-socket\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.015457 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.012365 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-workload-socket\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.015457 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.014813 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-workload-certs\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.018138 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.018083 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-istio-envoy\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.019378 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.019331 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196bcedf-fdfd-4469-9234-d281f907d53a-istio-podinfo\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.023080 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.023057 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stx8m\" (UniqueName: \"kubernetes.io/projected/196bcedf-fdfd-4469-9234-d281f907d53a-kube-api-access-stx8m\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.023561 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.023538 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196bcedf-fdfd-4469-9234-d281f907d53a-istio-token\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.172952 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.172922 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:46.465248 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.465217 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj"] Nov 02 17:57:46.466657 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:57:46.465592 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod196bcedf_fdfd_4469_9234_d281f907d53a.slice/crio-8334079d2c4a8b8e49d059f530a720b5fe64586a32f37d0c89f8bb67b9550d45 WatchSource:0}: Error finding container 8334079d2c4a8b8e49d059f530a720b5fe64586a32f37d0c89f8bb67b9550d45: Status 404 returned error can't find the container with id 8334079d2c4a8b8e49d059f530a720b5fe64586a32f37d0c89f8bb67b9550d45 Nov 02 17:57:46.468543 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.468508 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:46.468669 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.468576 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:46.468669 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.468606 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:57:46.481250 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.481226 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:57:46.482478 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:46.482458 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 17:57:47.493419 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:47.492084 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" event={"ID":"196bcedf-fdfd-4469-9234-d281f907d53a","Type":"ContainerStarted","Data":"a61f531e9ab0c910390c112acf6a8bf15a30f53e37c5ffabf070b84bdb9a8954"} Nov 02 17:57:47.493419 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:47.492152 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" event={"ID":"196bcedf-fdfd-4469-9234-d281f907d53a","Type":"ContainerStarted","Data":"8334079d2c4a8b8e49d059f530a720b5fe64586a32f37d0c89f8bb67b9550d45"} Nov 02 17:57:47.518540 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:47.518487 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" podStartSLOduration=2.518470963 podStartE2EDuration="2.518470963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:47.513280158 +0000 UTC m=+1432.204334171" watchObservedRunningTime="2026-11-02 17:57:47.518470963 +0000 UTC m=+1432.209524975" Nov 02 17:57:48.174135 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:48.174042 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:48.176193 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:48.176164 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 02 17:57:48.176361 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:48.176215 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" podUID="196bcedf-fdfd-4469-9234-d281f907d53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 02 17:57:49.173703 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:49.173671 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 02 17:57:49.174108 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:49.173725 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" podUID="196bcedf-fdfd-4469-9234-d281f907d53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 02 17:57:50.173335 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:50.173298 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 02 17:57:50.173516 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:50.173365 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" podUID="196bcedf-fdfd-4469-9234-d281f907d53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 02 17:57:51.174355 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:51.174322 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 02 17:57:51.174900 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:51.174380 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" podUID="196bcedf-fdfd-4469-9234-d281f907d53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 02 17:57:52.166637 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:52.166031 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp"] Nov 02 17:57:52.178209 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:52.178169 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 02 17:57:52.178791 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:52.178247 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" podUID="196bcedf-fdfd-4469-9234-d281f907d53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 02 17:57:52.178791 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:52.178502 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" podUID="62c1c9ac-b512-462a-961e-93ac960cde22" containerName="istio-proxy" containerID="cri-o://2a3d034061e431cb778d461f53b91f9fdc6cda3de8561f245df9a0baa26ab552" gracePeriod=30 Nov 02 17:57:53.222297 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:53.222260 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:53.224037 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:53.224000 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:53.226125 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:53.226097 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 17:57:57.465589 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.465518 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:57:57.483469 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.483433 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c1c9ac-b512-462a-961e-93ac960cde22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c1c9ac-b512-462a-961e-93ac960cde22-istiod-ca-cert\") pod \"62c1c9ac-b512-462a-961e-93ac960cde22\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " Nov 02 17:57:57.483697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.483547 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-workload-certs\") pod \"62c1c9ac-b512-462a-961e-93ac960cde22\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " Nov 02 17:57:57.483697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.483585 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-istio-envoy\") pod \"62c1c9ac-b512-462a-961e-93ac960cde22\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " Nov 02 17:57:57.483830 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.483701 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62c1c9ac-b512-462a-961e-93ac960cde22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c1c9ac-b512-462a-961e-93ac960cde22-istio-podinfo\") pod \"62c1c9ac-b512-462a-961e-93ac960cde22\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " Nov 02 17:57:57.483830 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.483751 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c1c9ac-b512-462a-961e-93ac960cde22-istio-token\" (UniqueName: \"kubernetes.io/projected/62c1c9ac-b512-462a-961e-93ac960cde22-istio-token\") pod \"62c1c9ac-b512-462a-961e-93ac960cde22\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " Nov 02 17:57:57.483940 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.483840 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c1c9ac-b512-462a-961e-93ac960cde22-istiod-ca-cert" pod "62c1c9ac-b512-462a-961e-93ac960cde22" (UID: "62c1c9ac-b512-462a-961e-93ac960cde22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:57.483940 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.483866 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-credential-socket\") pod \"62c1c9ac-b512-462a-961e-93ac960cde22\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " Nov 02 17:57:57.484045 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.483943 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-workload-socket\") pod \"62c1c9ac-b512-462a-961e-93ac960cde22\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " Nov 02 17:57:57.484045 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.483951 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-workload-certs" pod "62c1c9ac-b512-462a-961e-93ac960cde22" (UID: "62c1c9ac-b512-462a-961e-93ac960cde22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:57.484163 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.484037 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c1c9ac-b512-462a-961e-93ac960cde22-kube-api-access-4wfkf\" (UniqueName: \"kubernetes.io/projected/62c1c9ac-b512-462a-961e-93ac960cde22-kube-api-access-4wfkf\") pod \"62c1c9ac-b512-462a-961e-93ac960cde22\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " Nov 02 17:57:57.484163 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.484072 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-istio-data\") pod \"62c1c9ac-b512-462a-961e-93ac960cde22\" (UID: \"62c1c9ac-b512-462a-961e-93ac960cde22\") " Nov 02 17:57:57.484267 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.484168 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-workload-socket" pod "62c1c9ac-b512-462a-961e-93ac960cde22" (UID: "62c1c9ac-b512-462a-961e-93ac960cde22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:57.484267 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.484184 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-credential-socket" pod "62c1c9ac-b512-462a-961e-93ac960cde22" (UID: "62c1c9ac-b512-462a-961e-93ac960cde22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:57.484519 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.484494 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-istio-data" pod "62c1c9ac-b512-462a-961e-93ac960cde22" (UID: "62c1c9ac-b512-462a-961e-93ac960cde22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:57.484947 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.484887 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:57.485059 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.484949 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:57.485059 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.484965 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:57.485059 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.484979 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:57.485059 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.484988 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c1c9ac-b512-462a-961e-93ac960cde22-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:57.486380 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.486354 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-istio-envoy" pod "62c1c9ac-b512-462a-961e-93ac960cde22" (UID: "62c1c9ac-b512-462a-961e-93ac960cde22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:57.486561 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.486540 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62c1c9ac-b512-462a-961e-93ac960cde22-istio-podinfo" pod "62c1c9ac-b512-462a-961e-93ac960cde22" (UID: "62c1c9ac-b512-462a-961e-93ac960cde22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:57:57.486660 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.486562 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c1c9ac-b512-462a-961e-93ac960cde22-istio-token" pod "62c1c9ac-b512-462a-961e-93ac960cde22" (UID: "62c1c9ac-b512-462a-961e-93ac960cde22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:57.486660 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.486583 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c1c9ac-b512-462a-961e-93ac960cde22-kube-api-access-4wfkf" pod "62c1c9ac-b512-462a-961e-93ac960cde22" (UID: "62c1c9ac-b512-462a-961e-93ac960cde22"). InnerVolumeSpecName "kube-api-access-4wfkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:57.572711 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.572676 5601 generic.go:356] "Generic (PLEG): container finished" podID="62c1c9ac-b512-462a-961e-93ac960cde22" containerID="2a3d034061e431cb778d461f53b91f9fdc6cda3de8561f245df9a0baa26ab552" exitCode=0 Nov 02 17:57:57.572711 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.572721 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" Nov 02 17:57:57.572896 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.572760 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" event={"ID":"62c1c9ac-b512-462a-961e-93ac960cde22","Type":"ContainerDied","Data":"2a3d034061e431cb778d461f53b91f9fdc6cda3de8561f245df9a0baa26ab552"} Nov 02 17:57:57.572896 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.572819 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp" event={"ID":"62c1c9ac-b512-462a-961e-93ac960cde22","Type":"ContainerDied","Data":"658c555c084846031c76367302610fa8bba0ad443acd053d537705be7673b1ce"} Nov 02 17:57:57.572896 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.572848 5601 scope.go:122] "RemoveContainer" containerID="2a3d034061e431cb778d461f53b91f9fdc6cda3de8561f245df9a0baa26ab552" Nov 02 17:57:57.586097 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.586072 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wfkf\" (UniqueName: \"kubernetes.io/projected/62c1c9ac-b512-462a-961e-93ac960cde22-kube-api-access-4wfkf\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:57.586097 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.586097 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c1c9ac-b512-462a-961e-93ac960cde22-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:57.586251 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.586107 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c1c9ac-b512-462a-961e-93ac960cde22-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:57.586251 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.586116 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c1c9ac-b512-462a-961e-93ac960cde22-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:57.586586 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.586567 5601 scope.go:122] "RemoveContainer" containerID="2a3d034061e431cb778d461f53b91f9fdc6cda3de8561f245df9a0baa26ab552" Nov 02 17:57:57.586890 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:57:57.586870 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a3d034061e431cb778d461f53b91f9fdc6cda3de8561f245df9a0baa26ab552\": container with ID starting with 2a3d034061e431cb778d461f53b91f9fdc6cda3de8561f245df9a0baa26ab552 not found: ID does not exist" containerID="2a3d034061e431cb778d461f53b91f9fdc6cda3de8561f245df9a0baa26ab552" Nov 02 17:57:57.587000 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.586898 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a3d034061e431cb778d461f53b91f9fdc6cda3de8561f245df9a0baa26ab552"} err="failed to get container status \"2a3d034061e431cb778d461f53b91f9fdc6cda3de8561f245df9a0baa26ab552\": rpc error: code = NotFound desc = could not find container \"2a3d034061e431cb778d461f53b91f9fdc6cda3de8561f245df9a0baa26ab552\": container with ID starting with 2a3d034061e431cb778d461f53b91f9fdc6cda3de8561f245df9a0baa26ab552 not found: ID does not exist" Nov 02 17:57:57.614010 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.613816 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp"] Nov 02 17:57:57.618163 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:57.618131 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-jkjqp"] Nov 02 17:57:59.429072 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:57:59.429028 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c1c9ac-b512-462a-961e-93ac960cde22" path="/var/lib/kubelet/pods/62c1c9ac-b512-462a-961e-93ac960cde22/volumes" Nov 02 17:58:01.018264 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.018210 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2"] Nov 02 17:58:01.018969 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.018944 5601 cpu_manager.go:414] "containerMap: removing container" podUID="62c1c9ac-b512-462a-961e-93ac960cde22" containerName="istio-proxy" Nov 02 17:58:01.018969 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.018967 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c1c9ac-b512-462a-961e-93ac960cde22" containerName="istio-proxy" Nov 02 17:58:01.019146 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.019080 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c1c9ac-b512-462a-961e-93ac960cde22" containerName="istio-proxy" Nov 02 17:58:01.025860 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.025824 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.055759 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.037219 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2"] Nov 02 17:58:01.064957 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.064400 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.064957 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.064526 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.064957 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.064638 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.064957 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.064714 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.064957 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.064748 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.064957 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.064891 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.064957 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.064920 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb8zw\" (UniqueName: \"kubernetes.io/projected/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-kube-api-access-qb8zw\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.064957 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.064955 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.065552 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.065014 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.085638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.081675 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj"] Nov 02 17:58:01.123641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.114929 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.174644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.170959 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686404fe-e7bb-4a29-9f7f-e07021328d8c-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.174644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.171026 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.174644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.171054 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.174644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.171108 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.174644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.171178 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.174644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.171205 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.174644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.171254 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.174644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.171292 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.174644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.171356 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.174644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.171392 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.174644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.171464 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.174644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.171497 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.174644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.171671 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.174644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.171701 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb8zw\" (UniqueName: \"kubernetes.io/projected/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-kube-api-access-qb8zw\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.174644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.171737 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.174644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.171777 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.174644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.171829 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcnjz\" (UniqueName: \"kubernetes.io/projected/686404fe-e7bb-4a29-9f7f-e07021328d8c-kube-api-access-rcnjz\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.174644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.171864 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.174644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.172485 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.258436 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.208455 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj"] Nov 02 17:58:01.258436 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.209342 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.258436 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.229793 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.258436 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.255028 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.258436 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.255269 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.258436 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.255488 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.258436 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.256284 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb8zw\" (UniqueName: \"kubernetes.io/projected/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-kube-api-access-qb8zw\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.275918 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.275834 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.276193 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.276165 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcnjz\" (UniqueName: \"kubernetes.io/projected/686404fe-e7bb-4a29-9f7f-e07021328d8c-kube-api-access-rcnjz\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.276418 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.276399 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686404fe-e7bb-4a29-9f7f-e07021328d8c-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.276547 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.276531 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.308862 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.308822 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.317211 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.317180 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.317461 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.317442 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.317648 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.317609 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.317848 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.317831 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.321645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.290671 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686404fe-e7bb-4a29-9f7f-e07021328d8c-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.370641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.283156 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.370641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.306484 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.370641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.327897 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.370641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.317040 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.370641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.340413 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.370641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.340935 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.370641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.352250 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.370641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.355969 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.470778 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.423219 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.470778 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.433957 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcnjz\" (UniqueName: \"kubernetes.io/projected/686404fe-e7bb-4a29-9f7f-e07021328d8c-kube-api-access-rcnjz\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.535258 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.496055 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:01.535258 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.520558 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:01.732832 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.708497 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb"] Nov 02 17:58:01.875864 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.875837 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb"] Nov 02 17:58:01.876149 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:01.876121 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.012060 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.012009 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-data\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.019653 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.012534 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istiod-ca-cert\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.020395 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.020356 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zp6g\" (UniqueName: \"kubernetes.io/projected/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-kube-api-access-7zp6g\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.020642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.020604 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-credential-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.048813 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.048757 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-token\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.049246 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.049216 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-workload-certs\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.049444 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.049423 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-workload-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.049611 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.049567 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-envoy\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.049942 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.049919 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-podinfo\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.154129 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.154030 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-data\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.154407 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.154385 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istiod-ca-cert\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.154590 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.154572 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zp6g\" (UniqueName: \"kubernetes.io/projected/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-kube-api-access-7zp6g\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.154773 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.154754 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-credential-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.154916 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.154901 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-token\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.155055 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.155035 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-workload-certs\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.155189 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.155174 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-workload-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.155308 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.155293 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-envoy\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.155522 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.155505 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-podinfo\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.160596 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.160567 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-credential-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.171985 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.170652 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istiod-ca-cert\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.172241 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.170919 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-data\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.172639 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.171151 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-podinfo\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.172790 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.171492 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-workload-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.178794 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.178761 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-workload-certs\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.191068 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.191030 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-envoy\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.208742 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.203218 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zp6g\" (UniqueName: \"kubernetes.io/projected/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-kube-api-access-7zp6g\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.208742 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.203688 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-token\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.243688 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.235912 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:02.761493 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.759746 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2"] Nov 02 17:58:02.778269 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:58:02.778202 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode30e6af9_02a5_4b4f_900a_b1de3b3daae9.slice/crio-f46fa36f3897210d27a936c66bf1423cf701f49c27a1146d5a10b45419ab9435 WatchSource:0}: Error finding container f46fa36f3897210d27a936c66bf1423cf701f49c27a1146d5a10b45419ab9435: Status 404 returned error can't find the container with id f46fa36f3897210d27a936c66bf1423cf701f49c27a1146d5a10b45419ab9435 Nov 02 17:58:02.783580 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.783431 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:02.783580 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.783523 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:02.783580 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.783569 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:02.867857 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.867808 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" event={"ID":"e30e6af9-02a5-4b4f-900a-b1de3b3daae9","Type":"ContainerStarted","Data":"f46fa36f3897210d27a936c66bf1423cf701f49c27a1146d5a10b45419ab9435"} Nov 02 17:58:02.981375 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:58:02.981121 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37ac056_d4aa_4c98_b1c0_aeb3436786b2.slice/crio-a649fb2afa7ea1195ab33261fdbd93b3a9b240ed1a7e18241b60eb9f716b4a8d WatchSource:0}: Error finding container a649fb2afa7ea1195ab33261fdbd93b3a9b240ed1a7e18241b60eb9f716b4a8d: Status 404 returned error can't find the container with id a649fb2afa7ea1195ab33261fdbd93b3a9b240ed1a7e18241b60eb9f716b4a8d Nov 02 17:58:02.982691 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.982667 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb"] Nov 02 17:58:02.984337 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.984306 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:02.984410 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.984368 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:02.984410 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:02.984397 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:03.009095 ip-10-0-141-169 kubenswrapper[5601]: W1102 17:58:03.009060 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686404fe_e7bb_4a29_9f7f_e07021328d8c.slice/crio-f81f3f9f7ee9bbc065334ae84f8bf573afd2e2c291b3a0b4f67e91aca42a185c WatchSource:0}: Error finding container f81f3f9f7ee9bbc065334ae84f8bf573afd2e2c291b3a0b4f67e91aca42a185c: Status 404 returned error can't find the container with id f81f3f9f7ee9bbc065334ae84f8bf573afd2e2c291b3a0b4f67e91aca42a185c Nov 02 17:58:03.012296 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:03.011860 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj"] Nov 02 17:58:03.013031 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:03.012997 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:03.013145 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:03.013066 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:03.013145 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:03.013101 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:03.922203 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:03.922152 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" event={"ID":"e30e6af9-02a5-4b4f-900a-b1de3b3daae9","Type":"ContainerStarted","Data":"194b37023780e9409594a1e4c62849f7139dffa768b77538d52e75ffeb421a1c"} Nov 02 17:58:03.926540 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:03.926506 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" event={"ID":"686404fe-e7bb-4a29-9f7f-e07021328d8c","Type":"ContainerStarted","Data":"95370994daf483d00e49107340ad221f685658ac04fd8ea8b66ecece82bf9805"} Nov 02 17:58:03.926792 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:03.926777 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" event={"ID":"686404fe-e7bb-4a29-9f7f-e07021328d8c","Type":"ContainerStarted","Data":"f81f3f9f7ee9bbc065334ae84f8bf573afd2e2c291b3a0b4f67e91aca42a185c"} Nov 02 17:58:03.932993 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:03.932961 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" event={"ID":"b37ac056-d4aa-4c98-b1c0-aeb3436786b2","Type":"ContainerStarted","Data":"61aae5c336eb689d732e1af61c38aebc21d3f4ba0a6152393cdee38b301fc5c6"} Nov 02 17:58:03.933194 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:03.933181 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" event={"ID":"b37ac056-d4aa-4c98-b1c0-aeb3436786b2","Type":"ContainerStarted","Data":"a649fb2afa7ea1195ab33261fdbd93b3a9b240ed1a7e18241b60eb9f716b4a8d"} Nov 02 17:58:04.020408 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:04.020336 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" podStartSLOduration=4.020313273 podStartE2EDuration="4.020313273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:03.986854917 +0000 UTC m=+1448.677908965" watchObservedRunningTime="2026-11-02 17:58:04.020313273 +0000 UTC m=+1448.711367285" Nov 02 17:58:04.021830 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:04.021778 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" podStartSLOduration=3.021765355 podStartE2EDuration="3.021765355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:04.015716903 +0000 UTC m=+1448.706770915" watchObservedRunningTime="2026-11-02 17:58:04.021765355 +0000 UTC m=+1448.712819368" Nov 02 17:58:04.060649 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:04.047455 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" podStartSLOduration=3.047431476 podStartE2EDuration="3.047431476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:04.044864993 +0000 UTC m=+1448.735919007" watchObservedRunningTime="2026-11-02 17:58:04.047431476 +0000 UTC m=+1448.738485490" Nov 02 17:58:04.244698 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:04.238298 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:04.244698 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:04.238457 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 17:58:04.244698 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:04.238519 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" podUID="b37ac056-d4aa-4c98-b1c0-aeb3436786b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 17:58:04.537732 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:04.522065 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:04.537732 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:04.523373 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:04.551664 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:04.548603 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:58:04.551664 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:04.548682 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" podUID="686404fe-e7bb-4a29-9f7f-e07021328d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 17:58:04.551664 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:04.548973 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:58:04.551664 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:04.549003 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" podUID="e30e6af9-02a5-4b4f-900a-b1de3b3daae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 02 17:58:04.741166 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:58:04.741103 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:58:05.237071 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:05.237031 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 17:58:05.238036 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:05.237588 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" podUID="b37ac056-d4aa-4c98-b1c0-aeb3436786b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 17:58:05.499603 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:05.499105 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:58:05.499603 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:05.499165 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" podUID="686404fe-e7bb-4a29-9f7f-e07021328d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 17:58:05.526382 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:05.525764 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:58:05.526382 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:05.525829 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" podUID="e30e6af9-02a5-4b4f-900a-b1de3b3daae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 02 17:58:06.241083 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:06.241043 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 17:58:06.241824 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:06.241790 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" podUID="b37ac056-d4aa-4c98-b1c0-aeb3436786b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 17:58:06.503698 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:06.503546 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:58:06.503698 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:06.503632 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" podUID="686404fe-e7bb-4a29-9f7f-e07021328d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 17:58:06.525640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:06.524090 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:58:06.525640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:06.524172 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" podUID="e30e6af9-02a5-4b4f-900a-b1de3b3daae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 02 17:58:07.435744 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:07.406343 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 17:58:07.435744 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:07.407740 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" podUID="b37ac056-d4aa-4c98-b1c0-aeb3436786b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 17:58:07.558481 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:07.545137 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:58:07.655765 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:07.608668 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:07.656394 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:07.564152 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:58:07.656658 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:07.656627 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" podUID="e30e6af9-02a5-4b4f-900a-b1de3b3daae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 02 17:58:07.656810 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:07.640764 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" podUID="686404fe-e7bb-4a29-9f7f-e07021328d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 17:58:07.675838 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:07.640692 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:08.088640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.084745 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp"] Nov 02 17:58:08.120729 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.113829 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp"] Nov 02 17:58:08.120729 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.113950 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.120729 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.119881 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr"] Nov 02 17:58:08.139672 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.134855 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.190644 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.189781 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr"] Nov 02 17:58:08.240640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.236584 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 17:58:08.240640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.236657 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" podUID="b37ac056-d4aa-4c98-b1c0-aeb3436786b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 17:58:08.384409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.382664 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.384409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.382748 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.384409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.382786 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpr8f\" (UniqueName: \"kubernetes.io/projected/f398ebca-880e-42a8-bb69-9352537dddff-kube-api-access-rpr8f\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.384409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.382828 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.384409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.382860 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk4vb\" (UniqueName: \"kubernetes.io/projected/5ee859b4-abd7-4add-8511-1d4e2963e182-kube-api-access-xk4vb\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.384409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.382896 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.384409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.383022 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.384409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.383111 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.384409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.383160 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f398ebca-880e-42a8-bb69-9352537dddff-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.384409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.383218 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee859b4-abd7-4add-8511-1d4e2963e182-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.384409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.383243 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.384409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.383323 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f398ebca-880e-42a8-bb69-9352537dddff-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.384409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.383359 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.384409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.383400 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.384409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.383430 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.384409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.383484 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.384409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.383515 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.384409 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.383550 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f398ebca-880e-42a8-bb69-9352537dddff-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.488667 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.488763 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.488809 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f398ebca-880e-42a8-bb69-9352537dddff-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.488860 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee859b4-abd7-4add-8511-1d4e2963e182-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.488885 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.488950 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f398ebca-880e-42a8-bb69-9352537dddff-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.488986 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.489019 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.489049 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.489095 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.489124 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.489158 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f398ebca-880e-42a8-bb69-9352537dddff-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.489250 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.489307 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.489360 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpr8f\" (UniqueName: \"kubernetes.io/projected/f398ebca-880e-42a8-bb69-9352537dddff-kube-api-access-rpr8f\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.489408 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.489441 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk4vb\" (UniqueName: \"kubernetes.io/projected/5ee859b4-abd7-4add-8511-1d4e2963e182-kube-api-access-xk4vb\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.489474 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.494239 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.494487 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.506272 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.506536 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.507513 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee859b4-abd7-4add-8511-1d4e2963e182-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.509500 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.509547 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" podUID="686404fe-e7bb-4a29-9f7f-e07021328d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 17:58:08.580697 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.576316 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:08.607646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.590832 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.607646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.591107 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.607646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.591294 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.607646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.593803 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.607646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.593902 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:58:08.607646 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.593945 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" podUID="e30e6af9-02a5-4b4f-900a-b1de3b3daae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 02 17:58:08.649695 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.630215 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f398ebca-880e-42a8-bb69-9352537dddff-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.649695 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.644198 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:58:08.680647 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.654195 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f398ebca-880e-42a8-bb69-9352537dddff-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.725058 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.688718 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.727913 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.690172 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.728166 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.690341 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.728299 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.691163 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.728395 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.691701 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk4vb\" (UniqueName: \"kubernetes.io/projected/5ee859b4-abd7-4add-8511-1d4e2963e182-kube-api-access-xk4vb\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:08.728486 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.692254 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpr8f\" (UniqueName: \"kubernetes.io/projected/f398ebca-880e-42a8-bb69-9352537dddff-kube-api-access-rpr8f\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.728581 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.693764 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f398ebca-880e-42a8-bb69-9352537dddff-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:08.786641 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:08.785023 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:09.058663 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:09.058467 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:09.265638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:09.247072 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 17:58:09.265638 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:09.252754 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" podUID="b37ac056-d4aa-4c98-b1c0-aeb3436786b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 17:58:09.598642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:09.598528 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:58:09.598642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:09.598582 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" podUID="e30e6af9-02a5-4b4f-900a-b1de3b3daae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 02 17:58:09.630762 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:09.627858 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:58:09.630762 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:09.627915 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" podUID="686404fe-e7bb-4a29-9f7f-e07021328d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 17:58:09.905749 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:09.897683 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr"] Nov 02 17:58:09.906218 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:09.905103 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:09.906411 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:09.906385 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:09.906576 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:09.906553 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:10.202645 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:10.202075 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp"] Nov 02 17:58:10.205399 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:10.205339 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:10.205523 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:10.205465 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:10.205601 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:10.205528 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 17:58:10.208950 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:10.208913 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" event={"ID":"f398ebca-880e-42a8-bb69-9352537dddff","Type":"ContainerStarted","Data":"8228391266fe0b062292a0ba1a27742457edce3e470e947c662892013a266ad6"} Nov 02 17:58:10.246459 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:10.245714 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 17:58:10.246459 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:10.245774 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" podUID="b37ac056-d4aa-4c98-b1c0-aeb3436786b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 17:58:10.519695 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:10.515972 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:58:10.519695 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:10.516030 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" podUID="686404fe-e7bb-4a29-9f7f-e07021328d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 17:58:10.726642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:10.723041 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:10.754653 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:10.727371 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:10.754653 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:10.732600 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 17:58:11.276880 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:11.276849 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:11.278084 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:11.277905 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:11.281016 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:11.280939 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 17:58:11.281863 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:11.281834 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" event={"ID":"f398ebca-880e-42a8-bb69-9352537dddff","Type":"ContainerStarted","Data":"9bd937518ff1ddba8163957721784adf506596ef3cd25a33ce60b7c800f1be65"} Nov 02 17:58:11.285102 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:11.285076 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" event={"ID":"5ee859b4-abd7-4add-8511-1d4e2963e182","Type":"ContainerStarted","Data":"3a69163fac823f5a9926d1fabcac3ccaa3706a44d8c05cef3961ec2be3e0d4be"} Nov 02 17:58:11.285258 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:11.285113 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" event={"ID":"5ee859b4-abd7-4add-8511-1d4e2963e182","Type":"ContainerStarted","Data":"5a9d5ef7b81b8e418fc9db6f4cd4bb7cdca2dae3338959256a5893f9582d6e5a"} Nov 02 17:58:11.352164 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:11.352093 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" podStartSLOduration=3.352073228 podStartE2EDuration="3.352073228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:11.345523891 +0000 UTC m=+1456.036577905" watchObservedRunningTime="2026-11-02 17:58:11.352073228 +0000 UTC m=+1456.043127241" Nov 02 17:58:11.387169 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:11.386966 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" podStartSLOduration=3.386945783 podStartE2EDuration="3.386945783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:11.373294296 +0000 UTC m=+1456.064348322" watchObservedRunningTime="2026-11-02 17:58:11.386945783 +0000 UTC m=+1456.077999795" Nov 02 17:58:11.506420 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:11.506390 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:11.507360 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:11.507329 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:11.509073 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:11.509050 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 17:58:11.786743 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:11.786702 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:11.793667 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:11.792756 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 02 17:58:11.793667 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:11.792814 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" podUID="f398ebca-880e-42a8-bb69-9352537dddff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 02 17:58:12.062849 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:12.062684 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:12.063643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:12.063395 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 02 17:58:12.063643 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:12.063455 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" podUID="5ee859b4-abd7-4add-8511-1d4e2963e182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 02 17:58:12.790798 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:12.790759 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 02 17:58:12.791396 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:12.791324 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" podUID="f398ebca-880e-42a8-bb69-9352537dddff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 02 17:58:13.061500 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:13.061418 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 02 17:58:13.061840 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:13.061806 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" podUID="5ee859b4-abd7-4add-8511-1d4e2963e182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 02 17:58:13.787243 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:13.787204 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 02 17:58:13.788433 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:13.788398 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" podUID="f398ebca-880e-42a8-bb69-9352537dddff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 02 17:58:14.071746 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:14.065046 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 02 17:58:14.071746 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:14.065112 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" podUID="5ee859b4-abd7-4add-8511-1d4e2963e182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 02 17:58:14.792693 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:14.787034 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 02 17:58:14.792693 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:14.787102 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" podUID="f398ebca-880e-42a8-bb69-9352537dddff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 02 17:58:15.065855 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:15.064005 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 02 17:58:15.065855 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:15.064067 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" podUID="5ee859b4-abd7-4add-8511-1d4e2963e182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 02 17:58:15.791086 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:15.789142 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 02 17:58:15.791086 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:15.789208 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" podUID="f398ebca-880e-42a8-bb69-9352537dddff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 02 17:58:16.063702 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:16.063211 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 02 17:58:16.063702 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:16.063273 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" podUID="5ee859b4-abd7-4add-8511-1d4e2963e182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 02 17:58:16.786349 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:16.786310 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 02 17:58:16.786578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:16.786379 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" podUID="f398ebca-880e-42a8-bb69-9352537dddff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 02 17:58:17.068693 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:17.067025 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 02 17:58:17.068693 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:17.067100 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" podUID="5ee859b4-abd7-4add-8511-1d4e2963e182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 02 17:58:17.793460 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:17.793035 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:17.794573 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:17.794542 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:17.795526 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:17.795503 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 17:58:18.066642 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:18.066551 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:18.066996 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:18.066974 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:18.067747 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:18.067725 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 17:58:34.499860 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:58:34.499817 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:58:36.982632 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:36.982578 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7"] Nov 02 17:58:36.983796 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:36.983743 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" podUID="64fe7612-3ce2-451e-bcd1-0f80ebd1f352" containerName="maas-api" containerID="cri-o://3af6786929a709584db8f48a600250ba78b5a7e597bbd5633ec592d7b59fae07" gracePeriod=30 Nov 02 17:58:37.657156 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:37.656799 5601 generic.go:356] "Generic (PLEG): container finished" podID="64fe7612-3ce2-451e-bcd1-0f80ebd1f352" containerID="3af6786929a709584db8f48a600250ba78b5a7e597bbd5633ec592d7b59fae07" exitCode=0 Nov 02 17:58:37.657156 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:37.657065 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" event={"ID":"64fe7612-3ce2-451e-bcd1-0f80ebd1f352","Type":"ContainerDied","Data":"3af6786929a709584db8f48a600250ba78b5a7e597bbd5633ec592d7b59fae07"} Nov 02 17:58:37.702137 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:37.702111 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" Nov 02 17:58:37.770069 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:37.770039 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-maas-api-tls\") pod \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\" (UID: \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\") " Nov 02 17:58:37.770239 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:37.770165 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-kube-api-access-d46kb\" (UniqueName: \"kubernetes.io/projected/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-kube-api-access-d46kb\") pod \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\" (UID: \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\") " Nov 02 17:58:37.770239 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:37.770233 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-openshift-service-ca\") pod \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\" (UID: \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\") " Nov 02 17:58:37.770362 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:37.770347 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-metrics-certs\" (UniqueName: \"kubernetes.io/secret/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-metrics-certs\") pod \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\" (UID: \"64fe7612-3ce2-451e-bcd1-0f80ebd1f352\") " Nov 02 17:58:37.770705 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:37.770677 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-openshift-service-ca" pod "64fe7612-3ce2-451e-bcd1-0f80ebd1f352" (UID: "64fe7612-3ce2-451e-bcd1-0f80ebd1f352"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:37.771100 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:37.771079 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:37.772412 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:37.772388 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-kube-api-access-d46kb" pod "64fe7612-3ce2-451e-bcd1-0f80ebd1f352" (UID: "64fe7612-3ce2-451e-bcd1-0f80ebd1f352"). InnerVolumeSpecName "kube-api-access-d46kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:37.772722 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:37.772700 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-maas-api-tls" pod "64fe7612-3ce2-451e-bcd1-0f80ebd1f352" (UID: "64fe7612-3ce2-451e-bcd1-0f80ebd1f352"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:58:37.773556 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:37.773531 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-metrics-certs" pod "64fe7612-3ce2-451e-bcd1-0f80ebd1f352" (UID: "64fe7612-3ce2-451e-bcd1-0f80ebd1f352"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:58:37.899795 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:37.899756 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:37.899795 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:37.899794 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d46kb\" (UniqueName: \"kubernetes.io/projected/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-kube-api-access-d46kb\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:37.900029 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:37.899808 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64fe7612-3ce2-451e-bcd1-0f80ebd1f352-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:38.666233 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:38.666188 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" Nov 02 17:58:38.666765 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:38.666187 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7" event={"ID":"64fe7612-3ce2-451e-bcd1-0f80ebd1f352","Type":"ContainerDied","Data":"2c0adfc40205136ffc8c89d9c3456c1b8d9368bfda289848cd4f2c46315a15e6"} Nov 02 17:58:38.666765 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:38.666459 5601 scope.go:122] "RemoveContainer" containerID="3af6786929a709584db8f48a600250ba78b5a7e597bbd5633ec592d7b59fae07" Nov 02 17:58:38.708059 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:38.708023 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7"] Nov 02 17:58:38.714505 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:38.714473 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-23a657-5c6c9f9bf9-47bv7"] Nov 02 17:58:39.450733 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:39.450686 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64fe7612-3ce2-451e-bcd1-0f80ebd1f352" path="/var/lib/kubelet/pods/64fe7612-3ce2-451e-bcd1-0f80ebd1f352/volumes" Nov 02 17:58:43.927500 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:43.927464 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl"] Nov 02 17:58:43.928380 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:43.928353 5601 cpu_manager.go:414] "containerMap: removing container" podUID="64fe7612-3ce2-451e-bcd1-0f80ebd1f352" containerName="maas-api" Nov 02 17:58:43.928524 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:43.928504 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64fe7612-3ce2-451e-bcd1-0f80ebd1f352" containerName="maas-api" Nov 02 17:58:43.928712 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:43.928698 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="64fe7612-3ce2-451e-bcd1-0f80ebd1f352" containerName="maas-api" Nov 02 17:58:43.933705 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:43.933685 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:43.950508 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:43.950471 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-66d558\"/\"default-dockercfg-65hhm\"" Nov 02 17:58:43.950508 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:43.950472 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-66d558\"/\"openshift-service-ca.crt\"" Nov 02 17:58:43.950787 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:43.950603 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-66d558\"/\"noten-model-66d558-kserve-self-signed-certs\"" Nov 02 17:58:43.950787 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:43.950749 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-66d558\"/\"kube-root-ca.crt\"" Nov 02 17:58:43.953686 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:43.953596 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl"] Nov 02 17:58:43.999896 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:43.999857 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328d3cf6-d24a-4259-8919-4698fd1799b2-tls-certs\") pod \"noten-model-66d558-kserve-7887587b97-k58jl\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:43.999896 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:43.999900 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-model-cache\") pod \"noten-model-66d558-kserve-7887587b97-k58jl\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:44.000111 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.000033 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csh5z\" (UniqueName: \"kubernetes.io/projected/328d3cf6-d24a-4259-8919-4698fd1799b2-kube-api-access-csh5z\") pod \"noten-model-66d558-kserve-7887587b97-k58jl\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:44.000111 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.000092 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-dshm\") pod \"noten-model-66d558-kserve-7887587b97-k58jl\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:44.000239 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.000222 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-home\") pod \"noten-model-66d558-kserve-7887587b97-k58jl\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:44.000320 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.000306 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-tmp-dir\") pod \"noten-model-66d558-kserve-7887587b97-k58jl\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:44.108667 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.106711 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-dshm\") pod \"noten-model-66d558-kserve-7887587b97-k58jl\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:44.108667 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.106858 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-home\") pod \"noten-model-66d558-kserve-7887587b97-k58jl\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:44.108667 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.106967 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-tmp-dir\") pod \"noten-model-66d558-kserve-7887587b97-k58jl\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:44.108667 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.107064 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328d3cf6-d24a-4259-8919-4698fd1799b2-tls-certs\") pod \"noten-model-66d558-kserve-7887587b97-k58jl\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:44.108667 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.107102 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-model-cache\") pod \"noten-model-66d558-kserve-7887587b97-k58jl\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:44.108667 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.107201 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csh5z\" (UniqueName: \"kubernetes.io/projected/328d3cf6-d24a-4259-8919-4698fd1799b2-kube-api-access-csh5z\") pod \"noten-model-66d558-kserve-7887587b97-k58jl\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:44.113680 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.110191 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-tmp-dir\") pod \"noten-model-66d558-kserve-7887587b97-k58jl\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:44.113680 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.110444 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-home\") pod \"noten-model-66d558-kserve-7887587b97-k58jl\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:44.113680 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.110697 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-model-cache\") pod \"noten-model-66d558-kserve-7887587b97-k58jl\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:44.146637 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.145208 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csh5z\" (UniqueName: \"kubernetes.io/projected/328d3cf6-d24a-4259-8919-4698fd1799b2-kube-api-access-csh5z\") pod \"noten-model-66d558-kserve-7887587b97-k58jl\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:44.161141 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.161104 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-dshm\") pod \"noten-model-66d558-kserve-7887587b97-k58jl\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:44.180956 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.172297 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328d3cf6-d24a-4259-8919-4698fd1799b2-tls-certs\") pod \"noten-model-66d558-kserve-7887587b97-k58jl\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:44.275640 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.270950 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:44.643081 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.643025 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl"] Nov 02 17:58:44.754220 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.754178 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" event={"ID":"328d3cf6-d24a-4259-8919-4698fd1799b2","Type":"ContainerStarted","Data":"226108ce1e07c0c36f0784debb81dce635b28c65a49f596a231a7a73fd66271a"} Nov 02 17:58:44.754401 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:44.754234 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" event={"ID":"328d3cf6-d24a-4259-8919-4698fd1799b2","Type":"ContainerStarted","Data":"3482e083c694728a1b6b5b1d99876eeb981baaf718679e08ad1da774c4893883"} Nov 02 17:58:45.272218 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:45.272184 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:54.271678 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:54.271638 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:54.284891 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:54.284859 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:54.305953 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:54.305875 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" podStartSLOduration=11.305853506 podStartE2EDuration="11.305853506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:44.773858162 +0000 UTC m=+1489.464912177" watchObservedRunningTime="2026-11-02 17:58:54.305853506 +0000 UTC m=+1498.996907519" Nov 02 17:58:54.747941 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:54.747896 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl"] Nov 02 17:58:55.394904 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:58:55.394861 5601 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-self-signed-certs: secret "noten-model-66d558-kserve-self-signed-certs" not found Nov 02 17:58:55.401636 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:58:55.401596 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/328d3cf6-d24a-4259-8919-4698fd1799b2-tls-certs podName:328d3cf6-d24a-4259-8919-4698fd1799b2 nodeName:}" failed. No retries permitted until 2026-11-02 17:58:55.901575799 +0000 UTC m=+1500.592629803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/328d3cf6-d24a-4259-8919-4698fd1799b2-tls-certs") pod "noten-model-66d558-kserve-7887587b97-k58jl" (UID: "328d3cf6-d24a-4259-8919-4698fd1799b2") : secret "noten-model-66d558-kserve-self-signed-certs" not found Nov 02 17:58:55.402113 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:58:55.401222 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:58:55.903346 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:58:55.903313 5601 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-self-signed-certs: secret "noten-model-66d558-kserve-self-signed-certs" not found Nov 02 17:58:55.903508 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:58:55.903389 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/328d3cf6-d24a-4259-8919-4698fd1799b2-tls-certs podName:328d3cf6-d24a-4259-8919-4698fd1799b2 nodeName:}" failed. No retries permitted until 2026-11-02 17:58:56.903370749 +0000 UTC m=+1501.594424775 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/328d3cf6-d24a-4259-8919-4698fd1799b2-tls-certs") pod "noten-model-66d558-kserve-7887587b97-k58jl" (UID: "328d3cf6-d24a-4259-8919-4698fd1799b2") : secret "noten-model-66d558-kserve-self-signed-certs" not found Nov 02 17:58:56.929092 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:58:56.929051 5601 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-self-signed-certs: secret "noten-model-66d558-kserve-self-signed-certs" not found Nov 02 17:58:56.929494 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:58:56.929127 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/328d3cf6-d24a-4259-8919-4698fd1799b2-tls-certs podName:328d3cf6-d24a-4259-8919-4698fd1799b2 nodeName:}" failed. No retries permitted until 2026-11-02 17:58:58.929111925 +0000 UTC m=+1503.620165933 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/328d3cf6-d24a-4259-8919-4698fd1799b2-tls-certs") pod "noten-model-66d558-kserve-7887587b97-k58jl" (UID: "328d3cf6-d24a-4259-8919-4698fd1799b2") : secret "noten-model-66d558-kserve-self-signed-certs" not found Nov 02 17:58:58.966217 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:58:58.966183 5601 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-self-signed-certs: secret "noten-model-66d558-kserve-self-signed-certs" not found Nov 02 17:58:58.966612 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:58:58.966271 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/328d3cf6-d24a-4259-8919-4698fd1799b2-tls-certs podName:328d3cf6-d24a-4259-8919-4698fd1799b2 nodeName:}" failed. No retries permitted until 2026-11-02 17:59:02.966253201 +0000 UTC m=+1507.657307208 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/328d3cf6-d24a-4259-8919-4698fd1799b2-tls-certs") pod "noten-model-66d558-kserve-7887587b97-k58jl" (UID: "328d3cf6-d24a-4259-8919-4698fd1799b2") : secret "noten-model-66d558-kserve-self-signed-certs" not found Nov 02 17:59:03.057319 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:59:03.057280 5601 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-self-signed-certs: secret "noten-model-66d558-kserve-self-signed-certs" not found Nov 02 17:59:03.057743 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:59:03.057372 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/328d3cf6-d24a-4259-8919-4698fd1799b2-tls-certs podName:328d3cf6-d24a-4259-8919-4698fd1799b2 nodeName:}" failed. No retries permitted until 2026-11-02 17:59:11.057355029 +0000 UTC m=+1515.748409039 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/328d3cf6-d24a-4259-8919-4698fd1799b2-tls-certs") pod "noten-model-66d558-kserve-7887587b97-k58jl" (UID: "328d3cf6-d24a-4259-8919-4698fd1799b2") : secret "noten-model-66d558-kserve-self-signed-certs" not found Nov 02 17:59:04.496314 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:59:04.496275 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 17:59:11.121068 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:59:11.121031 5601 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-self-signed-certs: secret "noten-model-66d558-kserve-self-signed-certs" not found Nov 02 17:59:11.121554 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:59:11.121111 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/328d3cf6-d24a-4259-8919-4698fd1799b2-tls-certs podName:328d3cf6-d24a-4259-8919-4698fd1799b2 nodeName:}" failed. No retries permitted until 2026-11-02 17:59:27.12109331 +0000 UTC m=+1531.812147319 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/328d3cf6-d24a-4259-8919-4698fd1799b2-tls-certs") pod "noten-model-66d558-kserve-7887587b97-k58jl" (UID: "328d3cf6-d24a-4259-8919-4698fd1799b2") : secret "noten-model-66d558-kserve-self-signed-certs" not found Nov 02 17:59:11.320054 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.319990 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" podUID="328d3cf6-d24a-4259-8919-4698fd1799b2" containerName="main" containerID="cri-o://226108ce1e07c0c36f0784debb81dce635b28c65a49f596a231a7a73fd66271a" gracePeriod=45 Nov 02 17:59:11.438471 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.438439 5601 generic.go:356] "Generic (PLEG): container finished" podID="328d3cf6-d24a-4259-8919-4698fd1799b2" containerID="226108ce1e07c0c36f0784debb81dce635b28c65a49f596a231a7a73fd66271a" exitCode=0 Nov 02 17:59:11.438578 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.438530 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" event={"ID":"328d3cf6-d24a-4259-8919-4698fd1799b2","Type":"ContainerDied","Data":"226108ce1e07c0c36f0784debb81dce635b28c65a49f596a231a7a73fd66271a"} Nov 02 17:59:11.623380 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.623353 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:59:11.737047 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.736977 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-tmp-dir\") pod \"328d3cf6-d24a-4259-8919-4698fd1799b2\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " Nov 02 17:59:11.737047 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.737022 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/328d3cf6-d24a-4259-8919-4698fd1799b2-kube-api-access-csh5z\" (UniqueName: \"kubernetes.io/projected/328d3cf6-d24a-4259-8919-4698fd1799b2-kube-api-access-csh5z\") pod \"328d3cf6-d24a-4259-8919-4698fd1799b2\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " Nov 02 17:59:11.737281 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.737151 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-home\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-home\") pod \"328d3cf6-d24a-4259-8919-4698fd1799b2\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " Nov 02 17:59:11.737281 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.737185 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-dshm\") pod \"328d3cf6-d24a-4259-8919-4698fd1799b2\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " Nov 02 17:59:11.737403 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.737284 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-tmp-dir" pod "328d3cf6-d24a-4259-8919-4698fd1799b2" (UID: "328d3cf6-d24a-4259-8919-4698fd1799b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:11.737403 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.737298 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/328d3cf6-d24a-4259-8919-4698fd1799b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/328d3cf6-d24a-4259-8919-4698fd1799b2-tls-certs\") pod \"328d3cf6-d24a-4259-8919-4698fd1799b2\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " Nov 02 17:59:11.737518 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.737423 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-model-cache\") pod \"328d3cf6-d24a-4259-8919-4698fd1799b2\" (UID: \"328d3cf6-d24a-4259-8919-4698fd1799b2\") " Nov 02 17:59:11.737565 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.737425 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-home" pod "328d3cf6-d24a-4259-8919-4698fd1799b2" (UID: "328d3cf6-d24a-4259-8919-4698fd1799b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:11.737668 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.737630 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-model-cache" pod "328d3cf6-d24a-4259-8919-4698fd1799b2" (UID: "328d3cf6-d24a-4259-8919-4698fd1799b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:11.738444 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.738420 5601 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-tmp-dir\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:11.738542 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.738446 5601 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-home\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:11.738542 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.738461 5601 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-model-cache\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:11.739440 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.739418 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-dshm" pod "328d3cf6-d24a-4259-8919-4698fd1799b2" (UID: "328d3cf6-d24a-4259-8919-4698fd1799b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:11.739535 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.739460 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328d3cf6-d24a-4259-8919-4698fd1799b2-kube-api-access-csh5z" pod "328d3cf6-d24a-4259-8919-4698fd1799b2" (UID: "328d3cf6-d24a-4259-8919-4698fd1799b2"). InnerVolumeSpecName "kube-api-access-csh5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:59:11.739594 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.739534 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/328d3cf6-d24a-4259-8919-4698fd1799b2-tls-certs" pod "328d3cf6-d24a-4259-8919-4698fd1799b2" (UID: "328d3cf6-d24a-4259-8919-4698fd1799b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:59:11.840437 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.840408 5601 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/328d3cf6-d24a-4259-8919-4698fd1799b2-dshm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:11.840437 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.840433 5601 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328d3cf6-d24a-4259-8919-4698fd1799b2-tls-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:11.840437 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:11.840443 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csh5z\" (UniqueName: \"kubernetes.io/projected/328d3cf6-d24a-4259-8919-4698fd1799b2-kube-api-access-csh5z\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:12.448034 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:12.448007 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" Nov 02 17:59:12.448484 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:12.448006 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl" event={"ID":"328d3cf6-d24a-4259-8919-4698fd1799b2","Type":"ContainerDied","Data":"3482e083c694728a1b6b5b1d99876eeb981baaf718679e08ad1da774c4893883"} Nov 02 17:59:12.448484 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:12.448130 5601 scope.go:122] "RemoveContainer" containerID="226108ce1e07c0c36f0784debb81dce635b28c65a49f596a231a7a73fd66271a" Nov 02 17:59:12.475579 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:12.475547 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl"] Nov 02 17:59:12.478360 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:12.478337 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-66d558/noten-model-66d558-kserve-7887587b97-k58jl"] Nov 02 17:59:13.427022 ip-10-0-141-169 kubenswrapper[5601]: I1102 17:59:13.426989 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="328d3cf6-d24a-4259-8919-4698fd1799b2" path="/var/lib/kubelet/pods/328d3cf6-d24a-4259-8919-4698fd1799b2/volumes" Nov 02 17:59:34.486244 ip-10-0-141-169 kubenswrapper[5601]: E1102 17:59:34.486204 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:00:00.220338 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.220295 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t"] Nov 02 18:00:00.221028 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.221010 5601 cpu_manager.go:414] "containerMap: removing container" podUID="328d3cf6-d24a-4259-8919-4698fd1799b2" containerName="main" Nov 02 18:00:00.221075 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.221029 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="328d3cf6-d24a-4259-8919-4698fd1799b2" containerName="main" Nov 02 18:00:00.221172 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.221162 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="328d3cf6-d24a-4259-8919-4698fd1799b2" containerName="main" Nov 02 18:00:00.225658 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.225629 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" Nov 02 18:00:00.239847 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.239813 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv"] Nov 02 18:00:00.253224 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.253191 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" Nov 02 18:00:00.269331 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.269301 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t"] Nov 02 18:00:00.277140 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.277107 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv"] Nov 02 18:00:00.285662 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.285632 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s"] Nov 02 18:00:00.290167 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.290142 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k5dg\" (UniqueName: \"kubernetes.io/projected/69a5195a-7ccc-41ab-a5dc-e929d1043082-kube-api-access-2k5dg\") pod \"maas-api-key-cleanup-29894040-cvt7t\" (UID: \"69a5195a-7ccc-41ab-a5dc-e929d1043082\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" Nov 02 18:00:00.290305 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.290224 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qptzq\" (UniqueName: \"kubernetes.io/projected/6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd-kube-api-access-qptzq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv\" (UID: \"6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" Nov 02 18:00:00.290518 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.290488 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a5195a-7ccc-41ab-a5dc-e929d1043082-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894040-cvt7t\" (UID: \"69a5195a-7ccc-41ab-a5dc-e929d1043082\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" Nov 02 18:00:00.290727 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.290704 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv\" (UID: \"6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" Nov 02 18:00:00.291276 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.291257 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" Nov 02 18:00:00.298212 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.297099 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\"" Nov 02 18:00:00.298212 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.297427 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s"] Nov 02 18:00:00.392050 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.392015 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a5195a-7ccc-41ab-a5dc-e929d1043082-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894040-cvt7t\" (UID: \"69a5195a-7ccc-41ab-a5dc-e929d1043082\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" Nov 02 18:00:00.392244 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.392067 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj2gl\" (UniqueName: \"kubernetes.io/projected/84fe0745-0cf5-4be7-bbd6-dcf708f356d0-kube-api-access-qj2gl\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s\" (UID: \"84fe0745-0cf5-4be7-bbd6-dcf708f356d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" Nov 02 18:00:00.392244 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.392121 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv\" (UID: \"6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" Nov 02 18:00:00.392375 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.392263 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k5dg\" (UniqueName: \"kubernetes.io/projected/69a5195a-7ccc-41ab-a5dc-e929d1043082-kube-api-access-2k5dg\") pod \"maas-api-key-cleanup-29894040-cvt7t\" (UID: \"69a5195a-7ccc-41ab-a5dc-e929d1043082\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" Nov 02 18:00:00.392375 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.392325 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fe0745-0cf5-4be7-bbd6-dcf708f356d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s\" (UID: \"84fe0745-0cf5-4be7-bbd6-dcf708f356d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" Nov 02 18:00:00.392496 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.392382 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qptzq\" (UniqueName: \"kubernetes.io/projected/6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd-kube-api-access-qptzq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv\" (UID: \"6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" Nov 02 18:00:00.392886 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.392862 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a5195a-7ccc-41ab-a5dc-e929d1043082-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894040-cvt7t\" (UID: \"69a5195a-7ccc-41ab-a5dc-e929d1043082\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" Nov 02 18:00:00.392995 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.392868 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv\" (UID: \"6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" Nov 02 18:00:00.403419 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.403392 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qptzq\" (UniqueName: \"kubernetes.io/projected/6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd-kube-api-access-qptzq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv\" (UID: \"6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" Nov 02 18:00:00.403550 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.403432 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k5dg\" (UniqueName: \"kubernetes.io/projected/69a5195a-7ccc-41ab-a5dc-e929d1043082-kube-api-access-2k5dg\") pod \"maas-api-key-cleanup-29894040-cvt7t\" (UID: \"69a5195a-7ccc-41ab-a5dc-e929d1043082\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" Nov 02 18:00:00.493760 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.493679 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj2gl\" (UniqueName: \"kubernetes.io/projected/84fe0745-0cf5-4be7-bbd6-dcf708f356d0-kube-api-access-qj2gl\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s\" (UID: \"84fe0745-0cf5-4be7-bbd6-dcf708f356d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" Nov 02 18:00:00.493971 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.493937 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fe0745-0cf5-4be7-bbd6-dcf708f356d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s\" (UID: \"84fe0745-0cf5-4be7-bbd6-dcf708f356d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" Nov 02 18:00:00.494591 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.494569 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fe0745-0cf5-4be7-bbd6-dcf708f356d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s\" (UID: \"84fe0745-0cf5-4be7-bbd6-dcf708f356d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" Nov 02 18:00:00.503169 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.503147 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj2gl\" (UniqueName: \"kubernetes.io/projected/84fe0745-0cf5-4be7-bbd6-dcf708f356d0-kube-api-access-qj2gl\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s\" (UID: \"84fe0745-0cf5-4be7-bbd6-dcf708f356d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" Nov 02 18:00:00.574565 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.574527 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" Nov 02 18:00:00.586050 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.586023 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" Nov 02 18:00:00.611061 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.611034 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" Nov 02 18:00:00.650413 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.650374 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2"] Nov 02 18:00:00.668530 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.668168 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2"] Nov 02 18:00:00.668530 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.668268 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" Nov 02 18:00:00.695775 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.694071 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-0f0baf-serving-cert\"" Nov 02 18:00:00.701327 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.701293 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f3c081-92f7-4903-a974-779520012039-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2\" (UID: \"e5f3c081-92f7-4903-a974-779520012039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" Nov 02 18:00:00.701504 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.701343 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f3c081-92f7-4903-a974-779520012039-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2\" (UID: \"e5f3c081-92f7-4903-a974-779520012039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" Nov 02 18:00:00.701504 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.701454 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6gsq\" (UniqueName: \"kubernetes.io/projected/e5f3c081-92f7-4903-a974-779520012039-kube-api-access-z6gsq\") pod \"maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2\" (UID: \"e5f3c081-92f7-4903-a974-779520012039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" Nov 02 18:00:00.701709 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.701504 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f3c081-92f7-4903-a974-779520012039-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2\" (UID: \"e5f3c081-92f7-4903-a974-779520012039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" Nov 02 18:00:00.820882 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.820410 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6gsq\" (UniqueName: \"kubernetes.io/projected/e5f3c081-92f7-4903-a974-779520012039-kube-api-access-z6gsq\") pod \"maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2\" (UID: \"e5f3c081-92f7-4903-a974-779520012039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" Nov 02 18:00:00.820882 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.820489 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f3c081-92f7-4903-a974-779520012039-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2\" (UID: \"e5f3c081-92f7-4903-a974-779520012039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" Nov 02 18:00:00.827010 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.821307 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f3c081-92f7-4903-a974-779520012039-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2\" (UID: \"e5f3c081-92f7-4903-a974-779520012039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" Nov 02 18:00:00.827010 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.821352 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f3c081-92f7-4903-a974-779520012039-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2\" (UID: \"e5f3c081-92f7-4903-a974-779520012039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" Nov 02 18:00:00.827010 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.823147 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f3c081-92f7-4903-a974-779520012039-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2\" (UID: \"e5f3c081-92f7-4903-a974-779520012039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" Nov 02 18:00:00.836322 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.836257 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f3c081-92f7-4903-a974-779520012039-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2\" (UID: \"e5f3c081-92f7-4903-a974-779520012039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" Nov 02 18:00:00.854761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.854117 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f3c081-92f7-4903-a974-779520012039-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2\" (UID: \"e5f3c081-92f7-4903-a974-779520012039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" Nov 02 18:00:00.860045 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:00.859985 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6gsq\" (UniqueName: \"kubernetes.io/projected/e5f3c081-92f7-4903-a974-779520012039-kube-api-access-z6gsq\") pod \"maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2\" (UID: \"e5f3c081-92f7-4903-a974-779520012039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" Nov 02 18:00:01.031689 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:01.031611 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" Nov 02 18:00:01.120987 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:01.120920 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv"] Nov 02 18:00:01.127259 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:01.127227 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t"] Nov 02 18:00:01.185002 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:00:01.184954 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84fe0745_0cf5_4be7_bbd6_dcf708f356d0.slice/crio-fd8ae629bfdb44205920ac814e7cd9f47d446497d7cc4f192a39c887251d3ce1 WatchSource:0}: Error finding container fd8ae629bfdb44205920ac814e7cd9f47d446497d7cc4f192a39c887251d3ce1: Status 404 returned error can't find the container with id fd8ae629bfdb44205920ac814e7cd9f47d446497d7cc4f192a39c887251d3ce1 Nov 02 18:00:01.203811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:01.203780 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s"] Nov 02 18:00:01.333318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:01.333281 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2"] Nov 02 18:00:01.529730 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:01.526677 5601 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:00:02.103488 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:02.103379 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" event={"ID":"69a5195a-7ccc-41ab-a5dc-e929d1043082","Type":"ContainerStarted","Data":"02ff235332508040c1b6bac71e78611e1806040d8866749e895c4ac2b0436b9b"} Nov 02 18:00:02.103488 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:02.103448 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" event={"ID":"69a5195a-7ccc-41ab-a5dc-e929d1043082","Type":"ContainerStarted","Data":"a4114b6297494177d84642bf358cd15162ddffd645a24cbe54599e0730f66565"} Nov 02 18:00:02.105163 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:02.105130 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" event={"ID":"6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd","Type":"ContainerStarted","Data":"ed0b8f82a3eb48cfe322098405b59aef7fb3b3dd164460dc38109a64bbe8b43e"} Nov 02 18:00:02.105282 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:02.105178 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" event={"ID":"6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd","Type":"ContainerStarted","Data":"6732917f785296a8b08acfee68287ecec774884565a1dc6d923d22d6bbc4fd86"} Nov 02 18:00:02.106299 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:02.106274 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" event={"ID":"e5f3c081-92f7-4903-a974-779520012039","Type":"ContainerStarted","Data":"36e94d28ab9e1835fa261e870987167ca3602aece640c430854941d065f7ed90"} Nov 02 18:00:02.107807 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:02.107781 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" event={"ID":"84fe0745-0cf5-4be7-bbd6-dcf708f356d0","Type":"ContainerStarted","Data":"430ddb1f5f7c2dbab5f1b08fc68890ba2d6168ff35560e1924fcde58e011f507"} Nov 02 18:00:02.107924 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:02.107827 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" event={"ID":"84fe0745-0cf5-4be7-bbd6-dcf708f356d0","Type":"ContainerStarted","Data":"fd8ae629bfdb44205920ac814e7cd9f47d446497d7cc4f192a39c887251d3ce1"} Nov 02 18:00:02.126560 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:02.126503 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" podStartSLOduration=2.126488946 podStartE2EDuration="2.126488946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:02.124175634 +0000 UTC m=+1566.815229628" watchObservedRunningTime="2026-11-02 18:00:02.126488946 +0000 UTC m=+1566.817542963" Nov 02 18:00:02.145494 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:02.145428 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" podStartSLOduration=2.145409952 podStartE2EDuration="2.145409952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:02.143388031 +0000 UTC m=+1566.834442044" watchObservedRunningTime="2026-11-02 18:00:02.145409952 +0000 UTC m=+1566.836463965" Nov 02 18:00:02.165027 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:02.164976 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" podStartSLOduration=2.164961904 podStartE2EDuration="2.164961904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:02.162094471 +0000 UTC m=+1566.853148482" watchObservedRunningTime="2026-11-02 18:00:02.164961904 +0000 UTC m=+1566.856015915" Nov 02 18:00:03.117436 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:03.117396 5601 generic.go:356] "Generic (PLEG): container finished" podID="84fe0745-0cf5-4be7-bbd6-dcf708f356d0" containerID="430ddb1f5f7c2dbab5f1b08fc68890ba2d6168ff35560e1924fcde58e011f507" exitCode=60 Nov 02 18:00:03.117972 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:03.117484 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" event={"ID":"84fe0745-0cf5-4be7-bbd6-dcf708f356d0","Type":"ContainerDied","Data":"430ddb1f5f7c2dbab5f1b08fc68890ba2d6168ff35560e1924fcde58e011f507"} Nov 02 18:00:03.117972 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:03.117871 5601 scope.go:122] "RemoveContainer" containerID="430ddb1f5f7c2dbab5f1b08fc68890ba2d6168ff35560e1924fcde58e011f507" Nov 02 18:00:03.119378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:03.119357 5601 generic.go:356] "Generic (PLEG): container finished" podID="69a5195a-7ccc-41ab-a5dc-e929d1043082" containerID="02ff235332508040c1b6bac71e78611e1806040d8866749e895c4ac2b0436b9b" exitCode=60 Nov 02 18:00:03.119573 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:03.119460 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" event={"ID":"69a5195a-7ccc-41ab-a5dc-e929d1043082","Type":"ContainerDied","Data":"02ff235332508040c1b6bac71e78611e1806040d8866749e895c4ac2b0436b9b"} Nov 02 18:00:03.119748 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:03.119731 5601 scope.go:122] "RemoveContainer" containerID="02ff235332508040c1b6bac71e78611e1806040d8866749e895c4ac2b0436b9b" Nov 02 18:00:03.121521 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:03.121498 5601 generic.go:356] "Generic (PLEG): container finished" podID="6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd" containerID="ed0b8f82a3eb48cfe322098405b59aef7fb3b3dd164460dc38109a64bbe8b43e" exitCode=60 Nov 02 18:00:03.121676 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:03.121529 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" event={"ID":"6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd","Type":"ContainerDied","Data":"ed0b8f82a3eb48cfe322098405b59aef7fb3b3dd164460dc38109a64bbe8b43e"} Nov 02 18:00:03.121878 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:03.121818 5601 scope.go:122] "RemoveContainer" containerID="ed0b8f82a3eb48cfe322098405b59aef7fb3b3dd164460dc38109a64bbe8b43e" Nov 02 18:00:03.123691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:03.123672 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" event={"ID":"e5f3c081-92f7-4903-a974-779520012039","Type":"ContainerStarted","Data":"74859d646310f1000859f014cc10b72a75c05a7a507a07937eade4f6cc9f589d"} Nov 02 18:00:03.123820 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:03.123803 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" Nov 02 18:00:03.177959 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:03.177454 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" podStartSLOduration=2.230673073 podStartE2EDuration="3.177433491s" totalImagesPullingTime="946.760418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:03.168173297 +0000 UTC m=+1567.859227310" watchObservedRunningTime="2026-11-02 18:00:03.177433491 +0000 UTC m=+1567.868487505" Nov 02 18:00:04.132995 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:04.132961 5601 generic.go:356] "Generic (PLEG): container finished" podID="84fe0745-0cf5-4be7-bbd6-dcf708f356d0" containerID="36c188d8713248ca5f92b2754801ea702dfee298640a2dd375365b8025940b70" exitCode=60 Nov 02 18:00:04.133452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:04.133044 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" event={"ID":"84fe0745-0cf5-4be7-bbd6-dcf708f356d0","Type":"ContainerDied","Data":"36c188d8713248ca5f92b2754801ea702dfee298640a2dd375365b8025940b70"} Nov 02 18:00:04.133452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:04.133106 5601 scope.go:122] "RemoveContainer" containerID="430ddb1f5f7c2dbab5f1b08fc68890ba2d6168ff35560e1924fcde58e011f507" Nov 02 18:00:04.133452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:04.133433 5601 scope.go:122] "RemoveContainer" containerID="36c188d8713248ca5f92b2754801ea702dfee298640a2dd375365b8025940b70" Nov 02 18:00:04.133949 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:00:04.133907 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s_odh-ai-gateway-infra(84fe0745-0cf5-4be7-bbd6-dcf708f356d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" podUID="84fe0745-0cf5-4be7-bbd6-dcf708f356d0" Nov 02 18:00:04.135351 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:04.135322 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" event={"ID":"69a5195a-7ccc-41ab-a5dc-e929d1043082","Type":"ContainerStarted","Data":"abab75949ac2348b0c9ba66ed4e7dbafa87f680b1aaac9783f4acf7698100bfc"} Nov 02 18:00:04.137438 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:04.137413 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" event={"ID":"6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd","Type":"ContainerStarted","Data":"b6cdf82ef0f7e7f096f804f20f585974f55eaf2bb1c901ef9c2fab4f3a7c583f"} Nov 02 18:00:04.487685 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:00:04.487644 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:00:05.147609 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:05.147576 5601 scope.go:122] "RemoveContainer" containerID="36c188d8713248ca5f92b2754801ea702dfee298640a2dd375365b8025940b70" Nov 02 18:00:05.148218 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:00:05.148075 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s_odh-ai-gateway-infra(84fe0745-0cf5-4be7-bbd6-dcf708f356d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" podUID="84fe0745-0cf5-4be7-bbd6-dcf708f356d0" Nov 02 18:00:05.149868 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:05.149844 5601 generic.go:356] "Generic (PLEG): container finished" podID="69a5195a-7ccc-41ab-a5dc-e929d1043082" containerID="abab75949ac2348b0c9ba66ed4e7dbafa87f680b1aaac9783f4acf7698100bfc" exitCode=60 Nov 02 18:00:05.149961 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:05.149932 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" event={"ID":"69a5195a-7ccc-41ab-a5dc-e929d1043082","Type":"ContainerDied","Data":"abab75949ac2348b0c9ba66ed4e7dbafa87f680b1aaac9783f4acf7698100bfc"} Nov 02 18:00:05.150004 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:05.149995 5601 scope.go:122] "RemoveContainer" containerID="02ff235332508040c1b6bac71e78611e1806040d8866749e895c4ac2b0436b9b" Nov 02 18:00:05.150268 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:05.150246 5601 scope.go:122] "RemoveContainer" containerID="abab75949ac2348b0c9ba66ed4e7dbafa87f680b1aaac9783f4acf7698100bfc" Nov 02 18:00:05.150604 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:00:05.150582 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894040-cvt7t_odh-ai-gateway-infra(69a5195a-7ccc-41ab-a5dc-e929d1043082)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" podUID="69a5195a-7ccc-41ab-a5dc-e929d1043082" Nov 02 18:00:05.152239 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:05.152216 5601 generic.go:356] "Generic (PLEG): container finished" podID="6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd" containerID="b6cdf82ef0f7e7f096f804f20f585974f55eaf2bb1c901ef9c2fab4f3a7c583f" exitCode=60 Nov 02 18:00:05.152337 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:05.152299 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" event={"ID":"6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd","Type":"ContainerDied","Data":"b6cdf82ef0f7e7f096f804f20f585974f55eaf2bb1c901ef9c2fab4f3a7c583f"} Nov 02 18:00:05.152567 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:05.152554 5601 scope.go:122] "RemoveContainer" containerID="b6cdf82ef0f7e7f096f804f20f585974f55eaf2bb1c901ef9c2fab4f3a7c583f" Nov 02 18:00:05.152883 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:00:05.152864 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv_odh-ai-gateway-infra(6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" podUID="6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd" Nov 02 18:00:05.164095 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:05.164074 5601 scope.go:122] "RemoveContainer" containerID="ed0b8f82a3eb48cfe322098405b59aef7fb3b3dd164460dc38109a64bbe8b43e" Nov 02 18:00:06.161383 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:06.161350 5601 scope.go:122] "RemoveContainer" containerID="abab75949ac2348b0c9ba66ed4e7dbafa87f680b1aaac9783f4acf7698100bfc" Nov 02 18:00:06.161833 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:00:06.161797 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894040-cvt7t_odh-ai-gateway-infra(69a5195a-7ccc-41ab-a5dc-e929d1043082)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" podUID="69a5195a-7ccc-41ab-a5dc-e929d1043082" Nov 02 18:00:06.162891 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:06.162871 5601 scope.go:122] "RemoveContainer" containerID="b6cdf82ef0f7e7f096f804f20f585974f55eaf2bb1c901ef9c2fab4f3a7c583f" Nov 02 18:00:06.163230 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:00:06.163208 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv_odh-ai-gateway-infra(6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" podUID="6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd" Nov 02 18:00:09.143157 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:09.143125 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" Nov 02 18:00:15.423816 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:15.423784 5601 scope.go:122] "RemoveContainer" containerID="abab75949ac2348b0c9ba66ed4e7dbafa87f680b1aaac9783f4acf7698100bfc" Nov 02 18:00:15.424259 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:15.424186 5601 scope.go:122] "RemoveContainer" containerID="b6cdf82ef0f7e7f096f804f20f585974f55eaf2bb1c901ef9c2fab4f3a7c583f" Nov 02 18:00:16.267946 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:16.267905 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" event={"ID":"6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd","Type":"ContainerStarted","Data":"fc1cb5baf692e5450f43b329fa74f474cb197c74691a336a17282fb19bb33bf5"} Nov 02 18:00:16.269942 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:16.269918 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" event={"ID":"69a5195a-7ccc-41ab-a5dc-e929d1043082","Type":"ContainerStarted","Data":"9fc4a0779ab8c785b4104eb74616ab7198cc610ffcd83df5648f9ca267646934"} Nov 02 18:00:17.279442 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:17.279408 5601 generic.go:356] "Generic (PLEG): container finished" podID="69a5195a-7ccc-41ab-a5dc-e929d1043082" containerID="9fc4a0779ab8c785b4104eb74616ab7198cc610ffcd83df5648f9ca267646934" exitCode=60 Nov 02 18:00:17.279949 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:17.279478 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" event={"ID":"69a5195a-7ccc-41ab-a5dc-e929d1043082","Type":"ContainerDied","Data":"9fc4a0779ab8c785b4104eb74616ab7198cc610ffcd83df5648f9ca267646934"} Nov 02 18:00:17.279949 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:17.279545 5601 scope.go:122] "RemoveContainer" containerID="abab75949ac2348b0c9ba66ed4e7dbafa87f680b1aaac9783f4acf7698100bfc" Nov 02 18:00:17.280057 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:17.280040 5601 scope.go:122] "RemoveContainer" containerID="9fc4a0779ab8c785b4104eb74616ab7198cc610ffcd83df5648f9ca267646934" Nov 02 18:00:17.280840 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:00:17.280435 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894040-cvt7t_odh-ai-gateway-infra(69a5195a-7ccc-41ab-a5dc-e929d1043082)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" podUID="69a5195a-7ccc-41ab-a5dc-e929d1043082" Nov 02 18:00:17.281690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:17.281664 5601 generic.go:356] "Generic (PLEG): container finished" podID="6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd" containerID="fc1cb5baf692e5450f43b329fa74f474cb197c74691a336a17282fb19bb33bf5" exitCode=60 Nov 02 18:00:17.281774 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:17.281738 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" event={"ID":"6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd","Type":"ContainerDied","Data":"fc1cb5baf692e5450f43b329fa74f474cb197c74691a336a17282fb19bb33bf5"} Nov 02 18:00:17.282110 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:17.282094 5601 scope.go:122] "RemoveContainer" containerID="fc1cb5baf692e5450f43b329fa74f474cb197c74691a336a17282fb19bb33bf5" Nov 02 18:00:17.282383 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:00:17.282362 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv_odh-ai-gateway-infra(6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" podUID="6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd" Nov 02 18:00:17.293110 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:17.293089 5601 scope.go:122] "RemoveContainer" containerID="b6cdf82ef0f7e7f096f804f20f585974f55eaf2bb1c901ef9c2fab4f3a7c583f" Nov 02 18:00:17.306368 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:17.306335 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv"] Nov 02 18:00:17.324597 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:17.324563 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t"] Nov 02 18:00:18.419897 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:18.419862 5601 scope.go:122] "RemoveContainer" containerID="36c188d8713248ca5f92b2754801ea702dfee298640a2dd375365b8025940b70" Nov 02 18:00:18.684657 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:18.684628 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" Nov 02 18:00:18.687272 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:18.687248 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" Nov 02 18:00:18.806056 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:18.806017 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd-maas-api-service-ca\") pod \"6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd\" (UID: \"6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd\") " Nov 02 18:00:18.806309 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:18.806241 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a5195a-7ccc-41ab-a5dc-e929d1043082-kube-api-access-2k5dg\" (UniqueName: \"kubernetes.io/projected/69a5195a-7ccc-41ab-a5dc-e929d1043082-kube-api-access-2k5dg\") pod \"69a5195a-7ccc-41ab-a5dc-e929d1043082\" (UID: \"69a5195a-7ccc-41ab-a5dc-e929d1043082\") " Nov 02 18:00:18.806551 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:18.806524 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd-maas-api-service-ca" pod "6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd" (UID: "6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:18.806934 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:18.806910 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69a5195a-7ccc-41ab-a5dc-e929d1043082-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a5195a-7ccc-41ab-a5dc-e929d1043082-maas-api-service-ca\") pod \"69a5195a-7ccc-41ab-a5dc-e929d1043082\" (UID: \"69a5195a-7ccc-41ab-a5dc-e929d1043082\") " Nov 02 18:00:18.807032 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:18.807020 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd-kube-api-access-qptzq\" (UniqueName: \"kubernetes.io/projected/6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd-kube-api-access-qptzq\") pod \"6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd\" (UID: \"6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd\") " Nov 02 18:00:18.807304 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:18.807271 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a5195a-7ccc-41ab-a5dc-e929d1043082-maas-api-service-ca" pod "69a5195a-7ccc-41ab-a5dc-e929d1043082" (UID: "69a5195a-7ccc-41ab-a5dc-e929d1043082"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:18.807728 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:18.807708 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a5195a-7ccc-41ab-a5dc-e929d1043082-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.807728 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:18.807728 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.808826 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:18.808796 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a5195a-7ccc-41ab-a5dc-e929d1043082-kube-api-access-2k5dg" pod "69a5195a-7ccc-41ab-a5dc-e929d1043082" (UID: "69a5195a-7ccc-41ab-a5dc-e929d1043082"). InnerVolumeSpecName "kube-api-access-2k5dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:18.809143 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:18.809120 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd-kube-api-access-qptzq" pod "6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd" (UID: "6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd"). InnerVolumeSpecName "kube-api-access-qptzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:18.909361 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:18.909280 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qptzq\" (UniqueName: \"kubernetes.io/projected/6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd-kube-api-access-qptzq\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.909361 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:18.909324 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k5dg\" (UniqueName: \"kubernetes.io/projected/69a5195a-7ccc-41ab-a5dc-e929d1043082-kube-api-access-2k5dg\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:19.302097 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:19.302060 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" event={"ID":"84fe0745-0cf5-4be7-bbd6-dcf708f356d0","Type":"ContainerStarted","Data":"700f3e7223fbb16dfb38a64a51952a2fab287def9f54b2377d38ca1358b82029"} Nov 02 18:00:19.303728 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:19.303703 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" event={"ID":"69a5195a-7ccc-41ab-a5dc-e929d1043082","Type":"ContainerDied","Data":"a4114b6297494177d84642bf358cd15162ddffd645a24cbe54599e0730f66565"} Nov 02 18:00:19.303728 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:19.303719 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t" Nov 02 18:00:19.303927 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:19.303755 5601 scope.go:122] "RemoveContainer" containerID="9fc4a0779ab8c785b4104eb74616ab7198cc610ffcd83df5648f9ca267646934" Nov 02 18:00:19.305219 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:19.305199 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" Nov 02 18:00:19.305219 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:19.305207 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv" event={"ID":"6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd","Type":"ContainerDied","Data":"6732917f785296a8b08acfee68287ecec774884565a1dc6d923d22d6bbc4fd86"} Nov 02 18:00:19.314771 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:19.314753 5601 scope.go:122] "RemoveContainer" containerID="fc1cb5baf692e5450f43b329fa74f474cb197c74691a336a17282fb19bb33bf5" Nov 02 18:00:19.341699 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:19.341670 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t"] Nov 02 18:00:19.347314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:19.347287 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894040-cvt7t"] Nov 02 18:00:19.363080 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:19.363045 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv"] Nov 02 18:00:19.367464 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:19.367432 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894040-7zsgv"] Nov 02 18:00:19.429667 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:19.429636 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69a5195a-7ccc-41ab-a5dc-e929d1043082" path="/var/lib/kubelet/pods/69a5195a-7ccc-41ab-a5dc-e929d1043082/volumes" Nov 02 18:00:19.430284 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:19.430261 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd" path="/var/lib/kubelet/pods/6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd/volumes" Nov 02 18:00:20.318156 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:20.318107 5601 generic.go:356] "Generic (PLEG): container finished" podID="84fe0745-0cf5-4be7-bbd6-dcf708f356d0" containerID="700f3e7223fbb16dfb38a64a51952a2fab287def9f54b2377d38ca1358b82029" exitCode=60 Nov 02 18:00:20.318326 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:20.318158 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" event={"ID":"84fe0745-0cf5-4be7-bbd6-dcf708f356d0","Type":"ContainerDied","Data":"700f3e7223fbb16dfb38a64a51952a2fab287def9f54b2377d38ca1358b82029"} Nov 02 18:00:20.318326 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:20.318233 5601 scope.go:122] "RemoveContainer" containerID="36c188d8713248ca5f92b2754801ea702dfee298640a2dd375365b8025940b70" Nov 02 18:00:20.318766 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:20.318744 5601 scope.go:122] "RemoveContainer" containerID="700f3e7223fbb16dfb38a64a51952a2fab287def9f54b2377d38ca1358b82029" Nov 02 18:00:20.319173 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:00:20.319143 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s_odh-ai-gateway-infra(84fe0745-0cf5-4be7-bbd6-dcf708f356d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" podUID="84fe0745-0cf5-4be7-bbd6-dcf708f356d0" Nov 02 18:00:20.350970 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:20.350932 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s"] Nov 02 18:00:21.516869 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:21.516708 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" Nov 02 18:00:21.598199 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:21.598092 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84fe0745-0cf5-4be7-bbd6-dcf708f356d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fe0745-0cf5-4be7-bbd6-dcf708f356d0-maas-api-service-ca\") pod \"84fe0745-0cf5-4be7-bbd6-dcf708f356d0\" (UID: \"84fe0745-0cf5-4be7-bbd6-dcf708f356d0\") " Nov 02 18:00:21.598376 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:21.598320 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84fe0745-0cf5-4be7-bbd6-dcf708f356d0-kube-api-access-qj2gl\" (UniqueName: \"kubernetes.io/projected/84fe0745-0cf5-4be7-bbd6-dcf708f356d0-kube-api-access-qj2gl\") pod \"84fe0745-0cf5-4be7-bbd6-dcf708f356d0\" (UID: \"84fe0745-0cf5-4be7-bbd6-dcf708f356d0\") " Nov 02 18:00:21.599570 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:21.598514 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84fe0745-0cf5-4be7-bbd6-dcf708f356d0-maas-api-service-ca" pod "84fe0745-0cf5-4be7-bbd6-dcf708f356d0" (UID: "84fe0745-0cf5-4be7-bbd6-dcf708f356d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:21.599708 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:21.599681 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fe0745-0cf5-4be7-bbd6-dcf708f356d0-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:21.622875 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:21.622839 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84fe0745-0cf5-4be7-bbd6-dcf708f356d0-kube-api-access-qj2gl" pod "84fe0745-0cf5-4be7-bbd6-dcf708f356d0" (UID: "84fe0745-0cf5-4be7-bbd6-dcf708f356d0"). InnerVolumeSpecName "kube-api-access-qj2gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:21.701767 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:21.701733 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj2gl\" (UniqueName: \"kubernetes.io/projected/84fe0745-0cf5-4be7-bbd6-dcf708f356d0-kube-api-access-qj2gl\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:22.353032 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:22.352994 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" Nov 02 18:00:22.353032 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:22.353015 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s" event={"ID":"84fe0745-0cf5-4be7-bbd6-dcf708f356d0","Type":"ContainerDied","Data":"fd8ae629bfdb44205920ac814e7cd9f47d446497d7cc4f192a39c887251d3ce1"} Nov 02 18:00:22.353508 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:22.353091 5601 scope.go:122] "RemoveContainer" containerID="700f3e7223fbb16dfb38a64a51952a2fab287def9f54b2377d38ca1358b82029" Nov 02 18:00:22.392592 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:22.392527 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s"] Nov 02 18:00:22.418198 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:22.418156 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894040-chc9s"] Nov 02 18:00:23.427508 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:23.427474 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84fe0745-0cf5-4be7-bbd6-dcf708f356d0" path="/var/lib/kubelet/pods/84fe0745-0cf5-4be7-bbd6-dcf708f356d0/volumes" Nov 02 18:00:34.485772 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:00:34.485738 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:00:35.501504 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.501465 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs"] Nov 02 18:00:35.502167 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502151 5601 cpu_manager.go:414] "containerMap: removing container" podUID="84fe0745-0cf5-4be7-bbd6-dcf708f356d0" containerName="cleanup" Nov 02 18:00:35.502216 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502169 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fe0745-0cf5-4be7-bbd6-dcf708f356d0" containerName="cleanup" Nov 02 18:00:35.502216 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502180 5601 cpu_manager.go:414] "containerMap: removing container" podUID="84fe0745-0cf5-4be7-bbd6-dcf708f356d0" containerName="cleanup" Nov 02 18:00:35.502216 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502185 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fe0745-0cf5-4be7-bbd6-dcf708f356d0" containerName="cleanup" Nov 02 18:00:35.502216 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502196 5601 cpu_manager.go:414] "containerMap: removing container" podUID="69a5195a-7ccc-41ab-a5dc-e929d1043082" containerName="cleanup" Nov 02 18:00:35.502216 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502201 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69a5195a-7ccc-41ab-a5dc-e929d1043082" containerName="cleanup" Nov 02 18:00:35.502370 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502223 5601 cpu_manager.go:414] "containerMap: removing container" podUID="69a5195a-7ccc-41ab-a5dc-e929d1043082" containerName="cleanup" Nov 02 18:00:35.502370 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502228 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69a5195a-7ccc-41ab-a5dc-e929d1043082" containerName="cleanup" Nov 02 18:00:35.502370 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502251 5601 cpu_manager.go:414] "containerMap: removing container" podUID="6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd" containerName="cleanup" Nov 02 18:00:35.502370 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502256 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd" containerName="cleanup" Nov 02 18:00:35.502370 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502265 5601 cpu_manager.go:414] "containerMap: removing container" podUID="6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd" containerName="cleanup" Nov 02 18:00:35.502370 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502269 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd" containerName="cleanup" Nov 02 18:00:35.502370 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502280 5601 cpu_manager.go:414] "containerMap: removing container" podUID="6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd" containerName="cleanup" Nov 02 18:00:35.502370 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502284 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd" containerName="cleanup" Nov 02 18:00:35.502638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502386 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd" containerName="cleanup" Nov 02 18:00:35.502638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502400 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd" containerName="cleanup" Nov 02 18:00:35.502638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502414 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb8a08f-cba8-4820-82d8-ad8a9ebf71fd" containerName="cleanup" Nov 02 18:00:35.502638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502426 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fe0745-0cf5-4be7-bbd6-dcf708f356d0" containerName="cleanup" Nov 02 18:00:35.502638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502435 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fe0745-0cf5-4be7-bbd6-dcf708f356d0" containerName="cleanup" Nov 02 18:00:35.502638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502441 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="69a5195a-7ccc-41ab-a5dc-e929d1043082" containerName="cleanup" Nov 02 18:00:35.502638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502452 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="69a5195a-7ccc-41ab-a5dc-e929d1043082" containerName="cleanup" Nov 02 18:00:35.502638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.502463 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="69a5195a-7ccc-41ab-a5dc-e929d1043082" containerName="cleanup" Nov 02 18:00:35.507994 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.507963 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" Nov 02 18:00:35.519257 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.519222 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-72f605-serving-cert\"" Nov 02 18:00:35.530055 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.530026 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs"] Nov 02 18:00:35.687639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.682850 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44a346b2-4ddf-475b-9729-6e15d825198d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs\" (UID: \"44a346b2-4ddf-475b-9729-6e15d825198d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" Nov 02 18:00:35.687639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.682901 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a346b2-4ddf-475b-9729-6e15d825198d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs\" (UID: \"44a346b2-4ddf-475b-9729-6e15d825198d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" Nov 02 18:00:35.687639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.683049 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gt8m\" (UniqueName: \"kubernetes.io/projected/44a346b2-4ddf-475b-9729-6e15d825198d-kube-api-access-9gt8m\") pod \"maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs\" (UID: \"44a346b2-4ddf-475b-9729-6e15d825198d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" Nov 02 18:00:35.687639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.683078 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44a346b2-4ddf-475b-9729-6e15d825198d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs\" (UID: \"44a346b2-4ddf-475b-9729-6e15d825198d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" Nov 02 18:00:35.784421 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.784325 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gt8m\" (UniqueName: \"kubernetes.io/projected/44a346b2-4ddf-475b-9729-6e15d825198d-kube-api-access-9gt8m\") pod \"maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs\" (UID: \"44a346b2-4ddf-475b-9729-6e15d825198d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" Nov 02 18:00:35.784421 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.784365 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44a346b2-4ddf-475b-9729-6e15d825198d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs\" (UID: \"44a346b2-4ddf-475b-9729-6e15d825198d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" Nov 02 18:00:35.784703 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.784517 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44a346b2-4ddf-475b-9729-6e15d825198d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs\" (UID: \"44a346b2-4ddf-475b-9729-6e15d825198d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" Nov 02 18:00:35.784703 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.784539 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a346b2-4ddf-475b-9729-6e15d825198d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs\" (UID: \"44a346b2-4ddf-475b-9729-6e15d825198d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" Nov 02 18:00:35.784703 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:00:35.784687 5601 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-serving-cert: secret "maas-api-e2e-shared-b-w3-72f605-serving-cert" not found Nov 02 18:00:35.784851 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:00:35.784771 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a346b2-4ddf-475b-9729-6e15d825198d-maas-api-tls podName:44a346b2-4ddf-475b-9729-6e15d825198d nodeName:}" failed. No retries permitted until 2026-11-02 18:00:36.284754988 +0000 UTC m=+1600.975808982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/44a346b2-4ddf-475b-9729-6e15d825198d-maas-api-tls") pod "maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" (UID: "44a346b2-4ddf-475b-9729-6e15d825198d") : secret "maas-api-e2e-shared-b-w3-72f605-serving-cert" not found Nov 02 18:00:35.785130 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.785106 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44a346b2-4ddf-475b-9729-6e15d825198d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs\" (UID: \"44a346b2-4ddf-475b-9729-6e15d825198d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" Nov 02 18:00:35.786988 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.786957 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a346b2-4ddf-475b-9729-6e15d825198d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs\" (UID: \"44a346b2-4ddf-475b-9729-6e15d825198d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" Nov 02 18:00:35.793218 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:35.793198 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gt8m\" (UniqueName: \"kubernetes.io/projected/44a346b2-4ddf-475b-9729-6e15d825198d-kube-api-access-9gt8m\") pod \"maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs\" (UID: \"44a346b2-4ddf-475b-9729-6e15d825198d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" Nov 02 18:00:36.303639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:36.298774 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44a346b2-4ddf-475b-9729-6e15d825198d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs\" (UID: \"44a346b2-4ddf-475b-9729-6e15d825198d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" Nov 02 18:00:36.306689 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:36.306187 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44a346b2-4ddf-475b-9729-6e15d825198d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs\" (UID: \"44a346b2-4ddf-475b-9729-6e15d825198d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" Nov 02 18:00:36.437388 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:36.437356 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" Nov 02 18:00:36.724638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:36.724592 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs"] Nov 02 18:00:37.510636 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:37.509886 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" event={"ID":"44a346b2-4ddf-475b-9729-6e15d825198d","Type":"ContainerStarted","Data":"c89b85385248d06b13208691de04eacfef114e3ac310a94cfa2f76a655755d55"} Nov 02 18:00:37.510636 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:37.509963 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" Nov 02 18:00:37.510636 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:37.509995 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" event={"ID":"44a346b2-4ddf-475b-9729-6e15d825198d","Type":"ContainerStarted","Data":"19dd0a32babd23ce8d7ba32a01523cada321102c948fe064a256fef8968403f1"} Nov 02 18:00:37.530410 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:37.530349 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" podStartSLOduration=2.121910251 podStartE2EDuration="2.530333042s" totalImagesPullingTime="408.422791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:37.527987582 +0000 UTC m=+1602.219041601" watchObservedRunningTime="2026-11-02 18:00:37.530333042 +0000 UTC m=+1602.221387055" Nov 02 18:00:43.523508 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:00:43.523477 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" Nov 02 18:01:04.486169 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:01:04.486129 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:01:25.832232 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:25.832194 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8"] Nov 02 18:01:25.833038 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:25.833019 5601 cpu_manager.go:414] "containerMap: removing container" podUID="84fe0745-0cf5-4be7-bbd6-dcf708f356d0" containerName="cleanup" Nov 02 18:01:25.833122 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:25.833040 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fe0745-0cf5-4be7-bbd6-dcf708f356d0" containerName="cleanup" Nov 02 18:01:25.833122 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:25.833070 5601 cpu_manager.go:414] "containerMap: removing container" podUID="69a5195a-7ccc-41ab-a5dc-e929d1043082" containerName="cleanup" Nov 02 18:01:25.833122 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:25.833075 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69a5195a-7ccc-41ab-a5dc-e929d1043082" containerName="cleanup" Nov 02 18:01:25.833231 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:25.833199 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fe0745-0cf5-4be7-bbd6-dcf708f356d0" containerName="cleanup" Nov 02 18:01:25.837235 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:25.837210 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" Nov 02 18:01:25.848588 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:25.848556 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-5ec436-serving-cert\"" Nov 02 18:01:25.852807 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:25.852778 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8"] Nov 02 18:01:25.987685 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:25.987640 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f70b10-e5fb-4844-b55d-5883abbfd8e5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8\" (UID: \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" Nov 02 18:01:25.987864 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:25.987840 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13f70b10-e5fb-4844-b55d-5883abbfd8e5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8\" (UID: \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" Nov 02 18:01:25.987971 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:25.987951 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c8vp\" (UniqueName: \"kubernetes.io/projected/13f70b10-e5fb-4844-b55d-5883abbfd8e5-kube-api-access-8c8vp\") pod \"maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8\" (UID: \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" Nov 02 18:01:25.988089 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:25.988070 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13f70b10-e5fb-4844-b55d-5883abbfd8e5-metrics-certs\") pod \"maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8\" (UID: \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" Nov 02 18:01:26.089380 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:26.089298 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13f70b10-e5fb-4844-b55d-5883abbfd8e5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8\" (UID: \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" Nov 02 18:01:26.089380 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:26.089369 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c8vp\" (UniqueName: \"kubernetes.io/projected/13f70b10-e5fb-4844-b55d-5883abbfd8e5-kube-api-access-8c8vp\") pod \"maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8\" (UID: \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" Nov 02 18:01:26.089677 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:26.089428 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13f70b10-e5fb-4844-b55d-5883abbfd8e5-metrics-certs\") pod \"maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8\" (UID: \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" Nov 02 18:01:26.089677 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:26.089543 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f70b10-e5fb-4844-b55d-5883abbfd8e5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8\" (UID: \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" Nov 02 18:01:26.090269 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:26.090239 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f70b10-e5fb-4844-b55d-5883abbfd8e5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8\" (UID: \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" Nov 02 18:01:26.092027 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:26.091999 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13f70b10-e5fb-4844-b55d-5883abbfd8e5-metrics-certs\") pod \"maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8\" (UID: \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" Nov 02 18:01:26.092163 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:26.092140 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13f70b10-e5fb-4844-b55d-5883abbfd8e5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8\" (UID: \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" Nov 02 18:01:26.098731 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:26.098713 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c8vp\" (UniqueName: \"kubernetes.io/projected/13f70b10-e5fb-4844-b55d-5883abbfd8e5-kube-api-access-8c8vp\") pod \"maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8\" (UID: \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" Nov 02 18:01:26.176076 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:26.176040 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" Nov 02 18:01:26.548543 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:26.548016 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8"] Nov 02 18:01:26.550775 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:01:26.550448 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f70b10_e5fb_4844_b55d_5883abbfd8e5.slice/crio-079a7d0ff3039bf90e22c48dd8acab636f9385d7242e34aa855bccf20c4b186d WatchSource:0}: Error finding container 079a7d0ff3039bf90e22c48dd8acab636f9385d7242e34aa855bccf20c4b186d: Status 404 returned error can't find the container with id 079a7d0ff3039bf90e22c48dd8acab636f9385d7242e34aa855bccf20c4b186d Nov 02 18:01:26.850721 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:26.849987 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" event={"ID":"13f70b10-e5fb-4844-b55d-5883abbfd8e5","Type":"ContainerStarted","Data":"079a7d0ff3039bf90e22c48dd8acab636f9385d7242e34aa855bccf20c4b186d"} Nov 02 18:01:27.862061 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:27.862017 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" event={"ID":"13f70b10-e5fb-4844-b55d-5883abbfd8e5","Type":"ContainerStarted","Data":"5674dd33ce9268848c0406005e36739a44d0dd97ef676b20384cbfb5fb25bf9a"} Nov 02 18:01:27.862495 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:27.862135 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" Nov 02 18:01:27.887446 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:27.887398 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" podStartSLOduration=2.439481772 podStartE2EDuration="2.887385053s" totalImagesPullingTime="447.903281ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:27.883393282 +0000 UTC m=+1652.574447296" watchObservedRunningTime="2026-11-02 18:01:27.887385053 +0000 UTC m=+1652.578439066" Nov 02 18:01:34.032642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:34.027495 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" Nov 02 18:01:34.505003 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:01:34.504858 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:01:38.186659 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.182783 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9"] Nov 02 18:01:38.188594 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.188565 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.198426 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.198395 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-327608-openshift-default-dockercfg-8sqz6\"" Nov 02 18:01:38.203272 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.203245 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9"] Nov 02 18:01:38.239985 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.239856 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-credential-socket\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.239985 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.239950 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf577\" (UniqueName: \"kubernetes.io/projected/575ac171-83b1-4d32-bc19-9a039f885c0a-kube-api-access-bf577\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.240226 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.240194 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-workload-socket\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.240285 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.240235 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/575ac171-83b1-4d32-bc19-9a039f885c0a-istiod-ca-cert\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.240421 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.240399 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-podinfo\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.240498 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.240474 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-workload-certs\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.240607 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.240583 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-envoy\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.240709 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.240638 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-token\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.241193 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.240754 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-data\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.342886 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.342842 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf577\" (UniqueName: \"kubernetes.io/projected/575ac171-83b1-4d32-bc19-9a039f885c0a-kube-api-access-bf577\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.344368 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.344338 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-workload-socket\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.344995 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.344975 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/575ac171-83b1-4d32-bc19-9a039f885c0a-istiod-ca-cert\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.345270 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.345248 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-podinfo\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.345454 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.345438 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-workload-certs\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.347158 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.347114 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-envoy\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.347771 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.347742 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-token\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.347989 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.347968 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-data\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.348211 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.348192 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-credential-socket\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.348730 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.348708 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-credential-socket\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.348867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.346594 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/575ac171-83b1-4d32-bc19-9a039f885c0a-istiod-ca-cert\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.349292 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.346895 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-workload-certs\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.349292 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.344917 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-workload-socket\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.349829 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.349809 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-data\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.405317 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.405244 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-podinfo\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.406156 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.406099 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-envoy\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.411819 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.411742 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf577\" (UniqueName: \"kubernetes.io/projected/575ac171-83b1-4d32-bc19-9a039f885c0a-kube-api-access-bf577\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.422675 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.422151 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-token\") pod \"e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:38.513689 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:38.512901 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:39.027469 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:39.027390 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9"] Nov 02 18:01:39.034445 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:01:39.033090 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575ac171_83b1_4d32_bc19_9a039f885c0a.slice/crio-7a3d54e04b72d93d6298a7f41cb30e31a97e81d6dc79fda3063867351f937356 WatchSource:0}: Error finding container 7a3d54e04b72d93d6298a7f41cb30e31a97e81d6dc79fda3063867351f937356: Status 404 returned error can't find the container with id 7a3d54e04b72d93d6298a7f41cb30e31a97e81d6dc79fda3063867351f937356 Nov 02 18:01:39.037560 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:39.037519 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:01:39.037668 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:39.037610 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:01:39.037712 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:39.037665 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:01:39.231886 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:39.231842 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" event={"ID":"575ac171-83b1-4d32-bc19-9a039f885c0a","Type":"ContainerStarted","Data":"feeeaf737ec76220a88d7b95c0b56dbf3b06b18d343cfa406fd59269a1adf883"} Nov 02 18:01:39.240742 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:39.240702 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" event={"ID":"575ac171-83b1-4d32-bc19-9a039f885c0a","Type":"ContainerStarted","Data":"7a3d54e04b72d93d6298a7f41cb30e31a97e81d6dc79fda3063867351f937356"} Nov 02 18:01:39.321691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:39.315692 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" podStartSLOduration=1.3156696270000001 podStartE2EDuration="1.315669627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:39.286806491 +0000 UTC m=+1663.977860504" watchObservedRunningTime="2026-11-02 18:01:39.315669627 +0000 UTC m=+1664.006723643" Nov 02 18:01:39.523638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:39.518972 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:40.520252 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:40.520221 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:41.250110 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:41.250079 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:41.251403 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:41.251377 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:01:45.126644 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:45.123098 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf"] Nov 02 18:01:45.162519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:45.162487 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" Nov 02 18:01:45.173727 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:45.173698 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-a8e574-serving-cert\"" Nov 02 18:01:45.176988 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:45.176958 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf"] Nov 02 18:01:45.369683 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:45.368764 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519549a8-7b5f-46f6-9e58-428c162bbe63-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf\" (UID: \"519549a8-7b5f-46f6-9e58-428c162bbe63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" Nov 02 18:01:45.369683 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:45.368927 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519549a8-7b5f-46f6-9e58-428c162bbe63-metrics-certs\") pod \"maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf\" (UID: \"519549a8-7b5f-46f6-9e58-428c162bbe63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" Nov 02 18:01:45.369683 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:45.369037 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dplzv\" (UniqueName: \"kubernetes.io/projected/519549a8-7b5f-46f6-9e58-428c162bbe63-kube-api-access-dplzv\") pod \"maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf\" (UID: \"519549a8-7b5f-46f6-9e58-428c162bbe63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" Nov 02 18:01:45.369683 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:45.369058 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519549a8-7b5f-46f6-9e58-428c162bbe63-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf\" (UID: \"519549a8-7b5f-46f6-9e58-428c162bbe63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" Nov 02 18:01:45.471244 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:45.471187 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dplzv\" (UniqueName: \"kubernetes.io/projected/519549a8-7b5f-46f6-9e58-428c162bbe63-kube-api-access-dplzv\") pod \"maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf\" (UID: \"519549a8-7b5f-46f6-9e58-428c162bbe63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" Nov 02 18:01:45.471546 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:45.471513 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519549a8-7b5f-46f6-9e58-428c162bbe63-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf\" (UID: \"519549a8-7b5f-46f6-9e58-428c162bbe63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" Nov 02 18:01:45.472489 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:45.472450 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519549a8-7b5f-46f6-9e58-428c162bbe63-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf\" (UID: \"519549a8-7b5f-46f6-9e58-428c162bbe63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" Nov 02 18:01:45.473027 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:45.472997 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519549a8-7b5f-46f6-9e58-428c162bbe63-metrics-certs\") pod \"maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf\" (UID: \"519549a8-7b5f-46f6-9e58-428c162bbe63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" Nov 02 18:01:45.474824 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:01:45.474691 5601 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-serving-cert: secret "maas-api-e2e-extmod-a-a8e574-serving-cert" not found Nov 02 18:01:45.474824 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:01:45.474793 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/519549a8-7b5f-46f6-9e58-428c162bbe63-maas-api-tls podName:519549a8-7b5f-46f6-9e58-428c162bbe63 nodeName:}" failed. No retries permitted until 2026-11-02 18:01:45.974768745 +0000 UTC m=+1670.665822743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/519549a8-7b5f-46f6-9e58-428c162bbe63-maas-api-tls") pod "maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" (UID: "519549a8-7b5f-46f6-9e58-428c162bbe63") : secret "maas-api-e2e-extmod-a-a8e574-serving-cert" not found Nov 02 18:01:45.476143 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:45.476099 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519549a8-7b5f-46f6-9e58-428c162bbe63-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf\" (UID: \"519549a8-7b5f-46f6-9e58-428c162bbe63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" Nov 02 18:01:45.478137 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:45.478113 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519549a8-7b5f-46f6-9e58-428c162bbe63-metrics-certs\") pod \"maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf\" (UID: \"519549a8-7b5f-46f6-9e58-428c162bbe63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" Nov 02 18:01:45.498961 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:45.498924 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dplzv\" (UniqueName: \"kubernetes.io/projected/519549a8-7b5f-46f6-9e58-428c162bbe63-kube-api-access-dplzv\") pod \"maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf\" (UID: \"519549a8-7b5f-46f6-9e58-428c162bbe63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" Nov 02 18:01:45.983118 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:45.983082 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519549a8-7b5f-46f6-9e58-428c162bbe63-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf\" (UID: \"519549a8-7b5f-46f6-9e58-428c162bbe63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" Nov 02 18:01:45.985746 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:45.985713 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519549a8-7b5f-46f6-9e58-428c162bbe63-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf\" (UID: \"519549a8-7b5f-46f6-9e58-428c162bbe63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" Nov 02 18:01:46.123374 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:46.123338 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" Nov 02 18:01:46.500702 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:46.500671 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf"] Nov 02 18:01:46.502038 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:01:46.501998 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519549a8_7b5f_46f6_9e58_428c162bbe63.slice/crio-75d6736a67196d4bb740a33585b0ec455287fe5999744b9f5492735b8d2bdf05 WatchSource:0}: Error finding container 75d6736a67196d4bb740a33585b0ec455287fe5999744b9f5492735b8d2bdf05: Status 404 returned error can't find the container with id 75d6736a67196d4bb740a33585b0ec455287fe5999744b9f5492735b8d2bdf05 Nov 02 18:01:47.443547 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:47.443505 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" event={"ID":"519549a8-7b5f-46f6-9e58-428c162bbe63","Type":"ContainerStarted","Data":"1e8d78114f538231e03a451cc17ef30dc9a6c751ff055a2542a78f171c4d0717"} Nov 02 18:01:47.443757 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:47.443574 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" event={"ID":"519549a8-7b5f-46f6-9e58-428c162bbe63","Type":"ContainerStarted","Data":"75d6736a67196d4bb740a33585b0ec455287fe5999744b9f5492735b8d2bdf05"} Nov 02 18:01:47.443875 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:47.443856 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" Nov 02 18:01:47.466935 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:47.466877 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" podStartSLOduration=2.095326205 podStartE2EDuration="2.46686384s" totalImagesPullingTime="371.537635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:47.462197062 +0000 UTC m=+1672.153251073" watchObservedRunningTime="2026-11-02 18:01:47.46686384 +0000 UTC m=+1672.157917850" Nov 02 18:01:53.538735 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:53.538702 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" Nov 02 18:01:56.981007 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:56.980967 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m"] Nov 02 18:01:56.989134 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:56.989106 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:01:56.992268 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:56.992242 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m"] Nov 02 18:01:57.024058 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:57.024028 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-807c08-serving-cert\"" Nov 02 18:01:57.064724 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:57.064686 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7adb9d7e-827f-4573-a294-2681a27a79e0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m\" (UID: \"7adb9d7e-827f-4573-a294-2681a27a79e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:01:57.064894 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:57.064752 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-metrics-certs\") pod \"maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m\" (UID: \"7adb9d7e-827f-4573-a294-2681a27a79e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:01:57.064894 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:57.064834 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgqwm\" (UniqueName: \"kubernetes.io/projected/7adb9d7e-827f-4573-a294-2681a27a79e0-kube-api-access-fgqwm\") pod \"maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m\" (UID: \"7adb9d7e-827f-4573-a294-2681a27a79e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:01:57.064894 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:57.064869 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m\" (UID: \"7adb9d7e-827f-4573-a294-2681a27a79e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:01:57.169121 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:57.169075 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7adb9d7e-827f-4573-a294-2681a27a79e0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m\" (UID: \"7adb9d7e-827f-4573-a294-2681a27a79e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:01:57.169349 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:57.169146 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-metrics-certs\") pod \"maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m\" (UID: \"7adb9d7e-827f-4573-a294-2681a27a79e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:01:57.169349 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:57.169186 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgqwm\" (UniqueName: \"kubernetes.io/projected/7adb9d7e-827f-4573-a294-2681a27a79e0-kube-api-access-fgqwm\") pod \"maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m\" (UID: \"7adb9d7e-827f-4573-a294-2681a27a79e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:01:57.169349 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:57.169214 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m\" (UID: \"7adb9d7e-827f-4573-a294-2681a27a79e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:01:57.169796 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:01:57.169438 5601 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-serving-cert: secret "maas-api-e2e-extmod-a-807c08-serving-cert" not found Nov 02 18:01:57.169796 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:01:57.169520 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-maas-api-tls podName:7adb9d7e-827f-4573-a294-2681a27a79e0 nodeName:}" failed. No retries permitted until 2026-11-02 18:01:57.669496175 +0000 UTC m=+1682.360550182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-maas-api-tls") pod "maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" (UID: "7adb9d7e-827f-4573-a294-2681a27a79e0") : secret "maas-api-e2e-extmod-a-807c08-serving-cert" not found Nov 02 18:01:57.170064 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:57.170039 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7adb9d7e-827f-4573-a294-2681a27a79e0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m\" (UID: \"7adb9d7e-827f-4573-a294-2681a27a79e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:01:57.172811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:57.172786 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-metrics-certs\") pod \"maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m\" (UID: \"7adb9d7e-827f-4573-a294-2681a27a79e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:01:57.180788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:57.180742 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgqwm\" (UniqueName: \"kubernetes.io/projected/7adb9d7e-827f-4573-a294-2681a27a79e0-kube-api-access-fgqwm\") pod \"maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m\" (UID: \"7adb9d7e-827f-4573-a294-2681a27a79e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:01:57.689641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:57.682720 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m\" (UID: \"7adb9d7e-827f-4573-a294-2681a27a79e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:01:57.689641 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:01:57.683321 5601 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-serving-cert: secret "maas-api-e2e-extmod-a-807c08-serving-cert" not found Nov 02 18:01:57.689641 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:01:57.683401 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-maas-api-tls podName:7adb9d7e-827f-4573-a294-2681a27a79e0 nodeName:}" failed. No retries permitted until 2026-11-02 18:01:58.683381391 +0000 UTC m=+1683.374435384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-maas-api-tls") pod "maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" (UID: "7adb9d7e-827f-4573-a294-2681a27a79e0") : secret "maas-api-e2e-extmod-a-807c08-serving-cert" not found Nov 02 18:01:58.730637 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:58.729183 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m\" (UID: \"7adb9d7e-827f-4573-a294-2681a27a79e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:01:58.750506 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:58.750470 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m\" (UID: \"7adb9d7e-827f-4573-a294-2681a27a79e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:01:58.807515 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:58.807478 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:01:59.108252 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:01:59.108214 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7adb9d7e_827f_4573_a294_2681a27a79e0.slice/crio-0cac2eca335dca4ba5d9bed998eff96d9fea048837841bb1db5cdf9e271e5461 WatchSource:0}: Error finding container 0cac2eca335dca4ba5d9bed998eff96d9fea048837841bb1db5cdf9e271e5461: Status 404 returned error can't find the container with id 0cac2eca335dca4ba5d9bed998eff96d9fea048837841bb1db5cdf9e271e5461 Nov 02 18:01:59.109680 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:59.109651 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m"] Nov 02 18:01:59.922939 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:59.920478 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" event={"ID":"7adb9d7e-827f-4573-a294-2681a27a79e0","Type":"ContainerStarted","Data":"80e93daee116eb265fc28f553827d2074a9b84a20340589bc420202a50e520b8"} Nov 02 18:01:59.922939 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:59.920553 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" event={"ID":"7adb9d7e-827f-4573-a294-2681a27a79e0","Type":"ContainerStarted","Data":"0cac2eca335dca4ba5d9bed998eff96d9fea048837841bb1db5cdf9e271e5461"} Nov 02 18:01:59.938235 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:59.930008 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:01:59.957321 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:01:59.957268 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" podStartSLOduration=3.553868842 podStartE2EDuration="3.957252005s" totalImagesPullingTime="403.383163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:59.954226841 +0000 UTC m=+1684.645280853" watchObservedRunningTime="2026-11-02 18:01:59.957252005 +0000 UTC m=+1684.648306016" Nov 02 18:02:01.324924 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:01.324884 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t"] Nov 02 18:02:01.330325 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:01.330300 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" Nov 02 18:02:01.334323 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:01.334285 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-327608-serving-cert\"" Nov 02 18:02:01.338440 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:01.338405 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t"] Nov 02 18:02:01.417163 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:01.417127 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t\" (UID: \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" Nov 02 18:02:01.417353 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:01.417314 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-metrics-certs\") pod \"maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t\" (UID: \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" Nov 02 18:02:01.417465 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:01.417447 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t\" (UID: \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" Nov 02 18:02:01.417565 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:01.417477 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blhwc\" (UniqueName: \"kubernetes.io/projected/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-kube-api-access-blhwc\") pod \"maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t\" (UID: \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" Nov 02 18:02:01.533924 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:01.533789 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t\" (UID: \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" Nov 02 18:02:01.533924 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:01.533849 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blhwc\" (UniqueName: \"kubernetes.io/projected/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-kube-api-access-blhwc\") pod \"maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t\" (UID: \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" Nov 02 18:02:01.538996 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:01.534221 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t\" (UID: \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" Nov 02 18:02:01.538996 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:01.534416 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-metrics-certs\") pod \"maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t\" (UID: \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" Nov 02 18:02:01.545064 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:01.544789 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t\" (UID: \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" Nov 02 18:02:01.545931 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:02:01.545900 5601 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-serving-cert: secret "maas-api-e2e-extmod-b-327608-serving-cert" not found Nov 02 18:02:01.546080 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:02:01.545974 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-maas-api-tls podName:5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2 nodeName:}" failed. No retries permitted until 2026-11-02 18:02:02.045953304 +0000 UTC m=+1686.737007296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-maas-api-tls") pod "maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" (UID: "5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2") : secret "maas-api-e2e-extmod-b-327608-serving-cert" not found Nov 02 18:02:01.547045 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:01.547016 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-metrics-certs\") pod \"maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t\" (UID: \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" Nov 02 18:02:01.556848 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:01.556724 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blhwc\" (UniqueName: \"kubernetes.io/projected/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-kube-api-access-blhwc\") pod \"maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t\" (UID: \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" Nov 02 18:02:02.149553 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:02.148173 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t\" (UID: \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" Nov 02 18:02:02.188798 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:02.187588 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t\" (UID: \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" Nov 02 18:02:02.265970 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:02.265364 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" Nov 02 18:02:02.683259 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:02.683220 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t"] Nov 02 18:02:02.736109 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:02.736011 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69"] Nov 02 18:02:02.736341 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:02.736277 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" podUID="66c64f99-ea3f-4cef-bac8-372f00ea117c" containerName="istio-proxy" containerID="cri-o://cf4e2998e0a1bfb743eba102ca1d205b286d6e369397b8467476d55c181777b0" gracePeriod=30 Nov 02 18:02:02.946122 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:02.946087 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" event={"ID":"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2","Type":"ContainerStarted","Data":"038ee623bf1ade4a321abc486a356f9cdb435bcf937d1055e5a1970e546a3c8f"} Nov 02 18:02:03.955751 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:03.955708 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" event={"ID":"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2","Type":"ContainerStarted","Data":"de6d837ef31d7b7e3d22ef58192e15ee382da2f69737daafef445f7107ff0644"} Nov 02 18:02:03.956295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:03.955918 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" Nov 02 18:02:03.975876 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:03.975815 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" podStartSLOduration=2.56037179 podStartE2EDuration="2.975800311s" totalImagesPullingTime="415.428521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:03.973915956 +0000 UTC m=+1688.664969969" watchObservedRunningTime="2026-11-02 18:02:03.975800311 +0000 UTC m=+1688.666854322" Nov 02 18:02:04.489360 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:02:04.489324 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:02:06.943415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:06.943382 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:02:08.019512 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.019417 5601 generic.go:356] "Generic (PLEG): container finished" podID="66c64f99-ea3f-4cef-bac8-372f00ea117c" containerID="cf4e2998e0a1bfb743eba102ca1d205b286d6e369397b8467476d55c181777b0" exitCode=0 Nov 02 18:02:08.019512 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.019462 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" event={"ID":"66c64f99-ea3f-4cef-bac8-372f00ea117c","Type":"ContainerDied","Data":"cf4e2998e0a1bfb743eba102ca1d205b286d6e369397b8467476d55c181777b0"} Nov 02 18:02:08.547611 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.547538 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 18:02:08.640352 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.640301 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-token\" (UniqueName: \"kubernetes.io/projected/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-token\") pod \"66c64f99-ea3f-4cef-bac8-372f00ea117c\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " Nov 02 18:02:08.640563 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.640429 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-data\") pod \"66c64f99-ea3f-4cef-bac8-372f00ea117c\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " Nov 02 18:02:08.640563 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.640476 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-workload-certs\") pod \"66c64f99-ea3f-4cef-bac8-372f00ea117c\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " Nov 02 18:02:08.640563 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.640548 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c64f99-ea3f-4cef-bac8-372f00ea117c-kube-api-access-lkdsw\" (UniqueName: \"kubernetes.io/projected/66c64f99-ea3f-4cef-bac8-372f00ea117c-kube-api-access-lkdsw\") pod \"66c64f99-ea3f-4cef-bac8-372f00ea117c\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " Nov 02 18:02:08.640813 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.640708 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-podinfo\") pod \"66c64f99-ea3f-4cef-bac8-372f00ea117c\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " Nov 02 18:02:08.640813 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.640799 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-credential-socket\") pod \"66c64f99-ea3f-4cef-bac8-372f00ea117c\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " Nov 02 18:02:08.640920 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.640845 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-envoy\") pod \"66c64f99-ea3f-4cef-bac8-372f00ea117c\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " Nov 02 18:02:08.640973 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.640947 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-workload-socket\") pod \"66c64f99-ea3f-4cef-bac8-372f00ea117c\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " Nov 02 18:02:08.641030 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.641008 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66c64f99-ea3f-4cef-bac8-372f00ea117c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c64f99-ea3f-4cef-bac8-372f00ea117c-istiod-ca-cert\") pod \"66c64f99-ea3f-4cef-bac8-372f00ea117c\" (UID: \"66c64f99-ea3f-4cef-bac8-372f00ea117c\") " Nov 02 18:02:08.641889 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.641848 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-data" pod "66c64f99-ea3f-4cef-bac8-372f00ea117c" (UID: "66c64f99-ea3f-4cef-bac8-372f00ea117c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:08.642125 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.642100 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c64f99-ea3f-4cef-bac8-372f00ea117c-istiod-ca-cert" pod "66c64f99-ea3f-4cef-bac8-372f00ea117c" (UID: "66c64f99-ea3f-4cef-bac8-372f00ea117c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:02:08.642712 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.642690 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-workload-certs" pod "66c64f99-ea3f-4cef-bac8-372f00ea117c" (UID: "66c64f99-ea3f-4cef-bac8-372f00ea117c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:08.643551 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.643528 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-workload-socket" pod "66c64f99-ea3f-4cef-bac8-372f00ea117c" (UID: "66c64f99-ea3f-4cef-bac8-372f00ea117c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:08.644643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.644557 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-credential-socket" pod "66c64f99-ea3f-4cef-bac8-372f00ea117c" (UID: "66c64f99-ea3f-4cef-bac8-372f00ea117c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:08.648033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.648008 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-podinfo" pod "66c64f99-ea3f-4cef-bac8-372f00ea117c" (UID: "66c64f99-ea3f-4cef-bac8-372f00ea117c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:02:08.648193 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.648175 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c64f99-ea3f-4cef-bac8-372f00ea117c-kube-api-access-lkdsw" pod "66c64f99-ea3f-4cef-bac8-372f00ea117c" (UID: "66c64f99-ea3f-4cef-bac8-372f00ea117c"). InnerVolumeSpecName "kube-api-access-lkdsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:08.648412 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.648394 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-token" pod "66c64f99-ea3f-4cef-bac8-372f00ea117c" (UID: "66c64f99-ea3f-4cef-bac8-372f00ea117c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:08.650135 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.649520 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-envoy" pod "66c64f99-ea3f-4cef-bac8-372f00ea117c" (UID: "66c64f99-ea3f-4cef-bac8-372f00ea117c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:08.743627 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.743574 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:08.743627 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.743611 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:08.743627 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.743635 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c64f99-ea3f-4cef-bac8-372f00ea117c-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:08.743870 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.743646 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:08.743870 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.743655 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:08.743870 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.743664 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:08.743870 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.743674 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkdsw\" (UniqueName: \"kubernetes.io/projected/66c64f99-ea3f-4cef-bac8-372f00ea117c-kube-api-access-lkdsw\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:08.743870 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.743683 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c64f99-ea3f-4cef-bac8-372f00ea117c-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:08.743870 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.743691 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c64f99-ea3f-4cef-bac8-372f00ea117c-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:08.867611 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.867525 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9"] Nov 02 18:02:08.867823 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:08.867754 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" podUID="648e1353-2923-484a-835e-90b96e1acc50" containerName="istio-proxy" containerID="cri-o://2efea605288b4f806b659f00a712e7b83375159b042e45742587baf15c7c800a" gracePeriod=30 Nov 02 18:02:09.030804 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:09.030769 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" Nov 02 18:02:09.031236 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:09.030778 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69" event={"ID":"66c64f99-ea3f-4cef-bac8-372f00ea117c","Type":"ContainerDied","Data":"b718dc9d2392a0e7696d0239c4e83108d6e987fb306ca6ae7ff932d87f690bac"} Nov 02 18:02:09.031236 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:09.030909 5601 scope.go:122] "RemoveContainer" containerID="cf4e2998e0a1bfb743eba102ca1d205b286d6e369397b8467476d55c181777b0" Nov 02 18:02:09.061144 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:09.061095 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69"] Nov 02 18:02:09.065270 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:09.065244 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-ldf69"] Nov 02 18:02:09.428700 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:09.428666 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66c64f99-ea3f-4cef-bac8-372f00ea117c" path="/var/lib/kubelet/pods/66c64f99-ea3f-4cef-bac8-372f00ea117c/volumes" Nov 02 18:02:10.171640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:10.155201 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" Nov 02 18:02:12.823880 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:12.823842 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b"] Nov 02 18:02:12.824706 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:12.824686 5601 cpu_manager.go:414] "containerMap: removing container" podUID="66c64f99-ea3f-4cef-bac8-372f00ea117c" containerName="istio-proxy" Nov 02 18:02:12.824706 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:12.824706 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66c64f99-ea3f-4cef-bac8-372f00ea117c" containerName="istio-proxy" Nov 02 18:02:12.824905 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:12.824895 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="66c64f99-ea3f-4cef-bac8-372f00ea117c" containerName="istio-proxy" Nov 02 18:02:12.830238 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:12.830214 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:12.838697 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:12.838653 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-4ea582-openshift-default-dockercfg-4stc9\"" Nov 02 18:02:12.842005 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:12.841973 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b"] Nov 02 18:02:12.953959 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:12.953911 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-workload-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:12.954141 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:12.954002 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-workload-certs\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:12.954141 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:12.954045 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-data\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:12.954357 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:12.954330 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-token\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:12.954404 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:12.954392 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istiod-ca-cert\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:12.954501 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:12.954474 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-podinfo\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:12.954608 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:12.954594 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws5r4\" (UniqueName: \"kubernetes.io/projected/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-kube-api-access-ws5r4\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:12.954681 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:12.954659 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-envoy\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:12.954759 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:12.954738 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-credential-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.056240 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.056204 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-workload-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.056431 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.056334 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-workload-certs\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.056431 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.056385 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-data\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.056695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.056674 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-token\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.056783 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.056718 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-workload-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.056783 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.056726 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istiod-ca-cert\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.056898 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.056810 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-podinfo\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.056898 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.056878 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws5r4\" (UniqueName: \"kubernetes.io/projected/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-kube-api-access-ws5r4\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.057006 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.056901 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-envoy\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.057006 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.056937 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-credential-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.057457 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.057272 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-data\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.057457 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.057433 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-credential-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.057607 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.057462 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istiod-ca-cert\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.057607 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.057536 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-workload-certs\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.059811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.059779 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-podinfo\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.059924 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.059903 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-envoy\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.066082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.066058 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-token\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.066673 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.066602 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws5r4\" (UniqueName: \"kubernetes.io/projected/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-kube-api-access-ws5r4\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.149454 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.149365 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:13.466886 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.466552 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b"] Nov 02 18:02:13.473672 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:02:13.473524 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ee14b3_20d9_424b_beb3_211cb4ad7ae1.slice/crio-8383fd0c9f8a22aa3721976562b067dfbaf4b25529499f6966f95562bdaca643 WatchSource:0}: Error finding container 8383fd0c9f8a22aa3721976562b067dfbaf4b25529499f6966f95562bdaca643: Status 404 returned error can't find the container with id 8383fd0c9f8a22aa3721976562b067dfbaf4b25529499f6966f95562bdaca643 Nov 02 18:02:13.487798 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.481819 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:13.487798 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.481913 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:13.487798 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:13.481942 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:14.193741 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.193713 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 18:02:14.288757 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.288722 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648e1353-2923-484a-835e-90b96e1acc50-istio-token\" (UniqueName: \"kubernetes.io/projected/648e1353-2923-484a-835e-90b96e1acc50-istio-token\") pod \"648e1353-2923-484a-835e-90b96e1acc50\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " Nov 02 18:02:14.288958 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.288766 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-workload-socket\") pod \"648e1353-2923-484a-835e-90b96e1acc50\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " Nov 02 18:02:14.288958 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.288821 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/648e1353-2923-484a-835e-90b96e1acc50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648e1353-2923-484a-835e-90b96e1acc50-istio-podinfo\") pod \"648e1353-2923-484a-835e-90b96e1acc50\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " Nov 02 18:02:14.288958 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.288868 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/648e1353-2923-484a-835e-90b96e1acc50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648e1353-2923-484a-835e-90b96e1acc50-istiod-ca-cert\") pod \"648e1353-2923-484a-835e-90b96e1acc50\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " Nov 02 18:02:14.288958 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.288905 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-istio-data\") pod \"648e1353-2923-484a-835e-90b96e1acc50\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " Nov 02 18:02:14.289176 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.289024 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648e1353-2923-484a-835e-90b96e1acc50-kube-api-access-q7lp7\" (UniqueName: \"kubernetes.io/projected/648e1353-2923-484a-835e-90b96e1acc50-kube-api-access-q7lp7\") pod \"648e1353-2923-484a-835e-90b96e1acc50\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " Nov 02 18:02:14.289176 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.289078 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-credential-socket\") pod \"648e1353-2923-484a-835e-90b96e1acc50\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " Nov 02 18:02:14.289176 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.289106 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-istio-envoy\") pod \"648e1353-2923-484a-835e-90b96e1acc50\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " Nov 02 18:02:14.289336 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.289227 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-workload-socket" pod "648e1353-2923-484a-835e-90b96e1acc50" (UID: "648e1353-2923-484a-835e-90b96e1acc50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:14.289406 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.289384 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-credential-socket" pod "648e1353-2923-484a-835e-90b96e1acc50" (UID: "648e1353-2923-484a-835e-90b96e1acc50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:14.289462 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.289425 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648e1353-2923-484a-835e-90b96e1acc50-istiod-ca-cert" pod "648e1353-2923-484a-835e-90b96e1acc50" (UID: "648e1353-2923-484a-835e-90b96e1acc50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:02:14.289462 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.289453 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-istio-data" pod "648e1353-2923-484a-835e-90b96e1acc50" (UID: "648e1353-2923-484a-835e-90b96e1acc50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:14.289573 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.289195 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-workload-certs\") pod \"648e1353-2923-484a-835e-90b96e1acc50\" (UID: \"648e1353-2923-484a-835e-90b96e1acc50\") " Nov 02 18:02:14.289733 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.289713 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-workload-certs" pod "648e1353-2923-484a-835e-90b96e1acc50" (UID: "648e1353-2923-484a-835e-90b96e1acc50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:14.290388 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.290362 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:14.290388 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.290389 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:14.290567 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.290402 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:14.290567 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.290415 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648e1353-2923-484a-835e-90b96e1acc50-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:14.290567 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.290426 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:14.295843 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.292260 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-istio-envoy" pod "648e1353-2923-484a-835e-90b96e1acc50" (UID: "648e1353-2923-484a-835e-90b96e1acc50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:14.295843 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.292406 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/648e1353-2923-484a-835e-90b96e1acc50-istio-podinfo" pod "648e1353-2923-484a-835e-90b96e1acc50" (UID: "648e1353-2923-484a-835e-90b96e1acc50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:02:14.295843 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.295192 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648e1353-2923-484a-835e-90b96e1acc50-kube-api-access-q7lp7" pod "648e1353-2923-484a-835e-90b96e1acc50" (UID: "648e1353-2923-484a-835e-90b96e1acc50"). InnerVolumeSpecName "kube-api-access-q7lp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:14.295843 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.295449 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648e1353-2923-484a-835e-90b96e1acc50-istio-token" pod "648e1353-2923-484a-835e-90b96e1acc50" (UID: "648e1353-2923-484a-835e-90b96e1acc50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:14.394496 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.394444 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7lp7\" (UniqueName: \"kubernetes.io/projected/648e1353-2923-484a-835e-90b96e1acc50-kube-api-access-q7lp7\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:14.394496 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.394490 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648e1353-2923-484a-835e-90b96e1acc50-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:14.394785 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.394506 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648e1353-2923-484a-835e-90b96e1acc50-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:14.394785 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.394519 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648e1353-2923-484a-835e-90b96e1acc50-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:14.408922 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.408824 5601 generic.go:356] "Generic (PLEG): container finished" podID="648e1353-2923-484a-835e-90b96e1acc50" containerID="2efea605288b4f806b659f00a712e7b83375159b042e45742587baf15c7c800a" exitCode=0 Nov 02 18:02:14.408922 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.408887 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" Nov 02 18:02:14.409143 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.408916 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" event={"ID":"648e1353-2923-484a-835e-90b96e1acc50","Type":"ContainerDied","Data":"2efea605288b4f806b659f00a712e7b83375159b042e45742587baf15c7c800a"} Nov 02 18:02:14.409143 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.408984 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9" event={"ID":"648e1353-2923-484a-835e-90b96e1acc50","Type":"ContainerDied","Data":"80fef00ff834393adae309e695d29ef74d259fc29e4a6d520eebbf890a2c3f63"} Nov 02 18:02:14.409143 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.409014 5601 scope.go:122] "RemoveContainer" containerID="2efea605288b4f806b659f00a712e7b83375159b042e45742587baf15c7c800a" Nov 02 18:02:14.414264 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.414233 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" event={"ID":"09ee14b3-20d9-424b-beb3-211cb4ad7ae1","Type":"ContainerStarted","Data":"a8ffe84470fc33a5e11fc5acc0df077fe90bd345fe542f3e7e22c7cd3259e9f0"} Nov 02 18:02:14.414445 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.414290 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" event={"ID":"09ee14b3-20d9-424b-beb3-211cb4ad7ae1","Type":"ContainerStarted","Data":"8383fd0c9f8a22aa3721976562b067dfbaf4b25529499f6966f95562bdaca643"} Nov 02 18:02:14.428876 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.428847 5601 scope.go:122] "RemoveContainer" containerID="2efea605288b4f806b659f00a712e7b83375159b042e45742587baf15c7c800a" Nov 02 18:02:14.429363 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:02:14.429322 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2efea605288b4f806b659f00a712e7b83375159b042e45742587baf15c7c800a\": container with ID starting with 2efea605288b4f806b659f00a712e7b83375159b042e45742587baf15c7c800a not found: ID does not exist" containerID="2efea605288b4f806b659f00a712e7b83375159b042e45742587baf15c7c800a" Nov 02 18:02:14.429553 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.429371 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2efea605288b4f806b659f00a712e7b83375159b042e45742587baf15c7c800a"} err="failed to get container status \"2efea605288b4f806b659f00a712e7b83375159b042e45742587baf15c7c800a\": rpc error: code = NotFound desc = could not find container \"2efea605288b4f806b659f00a712e7b83375159b042e45742587baf15c7c800a\": container with ID starting with 2efea605288b4f806b659f00a712e7b83375159b042e45742587baf15c7c800a not found: ID does not exist" Nov 02 18:02:14.444690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.440691 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" podStartSLOduration=2.440673282 podStartE2EDuration="2.440673282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:14.436908219 +0000 UTC m=+1699.127962231" watchObservedRunningTime="2026-11-02 18:02:14.440673282 +0000 UTC m=+1699.131727294" Nov 02 18:02:14.461468 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.461436 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9"] Nov 02 18:02:14.466065 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:14.466031 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-5kmr9"] Nov 02 18:02:15.159494 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:15.159439 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:15.163059 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:15.162849 5601 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 18:02:15.163059 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:15.162910 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" podUID="09ee14b3-20d9-424b-beb3-211cb4ad7ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 18:02:15.632904 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:15.632873 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="648e1353-2923-484a-835e-90b96e1acc50" path="/var/lib/kubelet/pods/648e1353-2923-484a-835e-90b96e1acc50/volumes" Nov 02 18:02:16.154640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:16.153073 5601 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 18:02:16.154640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:16.153145 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" podUID="09ee14b3-20d9-424b-beb3-211cb4ad7ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 18:02:17.151640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:17.151027 5601 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 18:02:17.151640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:17.151093 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" podUID="09ee14b3-20d9-424b-beb3-211cb4ad7ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 18:02:18.161826 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:18.161497 5601 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 18:02:18.161826 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:18.161570 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" podUID="09ee14b3-20d9-424b-beb3-211cb4ad7ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 18:02:19.170756 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:19.156278 5601 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 18:02:19.170756 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:19.156342 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" podUID="09ee14b3-20d9-424b-beb3-211cb4ad7ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 18:02:20.159783 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.159739 5601 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 18:02:20.159976 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.159812 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" podUID="09ee14b3-20d9-424b-beb3-211cb4ad7ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 18:02:20.775475 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.775431 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj"] Nov 02 18:02:20.776519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.776492 5601 cpu_manager.go:414] "containerMap: removing container" podUID="648e1353-2923-484a-835e-90b96e1acc50" containerName="istio-proxy" Nov 02 18:02:20.776519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.776518 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="648e1353-2923-484a-835e-90b96e1acc50" containerName="istio-proxy" Nov 02 18:02:20.776803 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.776788 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="648e1353-2923-484a-835e-90b96e1acc50" containerName="istio-proxy" Nov 02 18:02:20.781013 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.780989 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.793003 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.792970 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-5ec436-openshift-default-dockercfg-wqbgt\"" Nov 02 18:02:20.800061 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.797823 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj"] Nov 02 18:02:20.870085 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.870044 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.870322 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.870293 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.870502 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.870473 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.870782 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.870750 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.870919 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.870897 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.870989 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.870930 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.870989 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.870965 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.871173 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.871057 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.871173 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.871112 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8fbs\" (UniqueName: \"kubernetes.io/projected/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-kube-api-access-z8fbs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.973005 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.972963 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.973005 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.973005 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.973280 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.973030 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.973280 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.973067 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.973280 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.973083 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8fbs\" (UniqueName: \"kubernetes.io/projected/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-kube-api-access-z8fbs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.973280 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.973196 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.973280 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.973218 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.973542 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.973315 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.973542 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.973394 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.974462 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.973945 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.974706 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.974489 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.974706 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.974554 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.974986 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.974957 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.975199 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.975176 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.976459 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.976437 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.977955 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.977912 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.982343 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.982319 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:20.982801 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:20.982784 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8fbs\" (UniqueName: \"kubernetes.io/projected/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-kube-api-access-z8fbs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:21.108502 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:21.108406 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:21.156336 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:21.156305 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:21.156815 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:21.156766 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:21.157580 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:21.157559 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:02:21.412325 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:02:21.412288 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d8a50a_572b_48b7_8c69_93d75c6c78eb.slice/crio-1fdf6f8a46d74c27e2418a7e9b730e3551e6808be29973b1e17f74936b0cf42f WatchSource:0}: Error finding container 1fdf6f8a46d74c27e2418a7e9b730e3551e6808be29973b1e17f74936b0cf42f: Status 404 returned error can't find the container with id 1fdf6f8a46d74c27e2418a7e9b730e3551e6808be29973b1e17f74936b0cf42f Nov 02 18:02:21.413973 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:21.413943 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj"] Nov 02 18:02:21.416129 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:21.415993 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:21.416129 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:21.416085 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:21.416302 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:21.416141 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:21.822422 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:21.822384 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" event={"ID":"b7d8a50a-572b-48b7-8c69-93d75c6c78eb","Type":"ContainerStarted","Data":"857937fb5820e416f86f7cc96de9ac48e12f4a5d632bbd6ca10320a6ce9c748c"} Nov 02 18:02:21.823549 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:21.823503 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" event={"ID":"b7d8a50a-572b-48b7-8c69-93d75c6c78eb","Type":"ContainerStarted","Data":"1fdf6f8a46d74c27e2418a7e9b730e3551e6808be29973b1e17f74936b0cf42f"} Nov 02 18:02:21.853434 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:21.853362 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" podStartSLOduration=1.853338648 podStartE2EDuration="1.853338648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:21.845362993 +0000 UTC m=+1706.536417007" watchObservedRunningTime="2026-11-02 18:02:21.853338648 +0000 UTC m=+1706.544392669" Nov 02 18:02:22.109050 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:22.108957 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:22.110631 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:22.110586 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 18:02:22.110763 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:22.110678 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" podUID="b7d8a50a-572b-48b7-8c69-93d75c6c78eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 18:02:23.108964 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:23.108928 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 18:02:23.109349 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:23.108996 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" podUID="b7d8a50a-572b-48b7-8c69-93d75c6c78eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 18:02:24.109703 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:24.109661 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 18:02:24.110163 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:24.109733 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" podUID="b7d8a50a-572b-48b7-8c69-93d75c6c78eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 18:02:25.177955 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:25.177915 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 18:02:25.179071 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:25.179037 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" podUID="b7d8a50a-572b-48b7-8c69-93d75c6c78eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 18:02:26.109199 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:26.109163 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 18:02:26.109421 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:26.109238 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" podUID="b7d8a50a-572b-48b7-8c69-93d75c6c78eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 18:02:27.109284 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:27.109249 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 18:02:27.109703 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:27.109301 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" podUID="b7d8a50a-572b-48b7-8c69-93d75c6c78eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 18:02:28.109295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:28.109262 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 18:02:28.109781 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:28.109315 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" podUID="b7d8a50a-572b-48b7-8c69-93d75c6c78eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 18:02:29.145691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:29.137000 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:29.145691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:29.139275 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:29.159644 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:29.153742 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:02:30.527867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:30.527822 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 02 18:02:30.663786 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:30.663740 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 18:02:30.924842 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:30.689692 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 18:02:30.925416 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:30.689736 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 18:02:30.926103 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:30.825086 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 18:02:34.498266 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:02:34.498221 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:02:41.251492 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:41.251407 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 18:02:41.251492 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:41.251476 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" podUID="575ac171-83b1-4d32-bc19-9a039f885c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 18:02:41.641447 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:41.641413 5601 generic.go:356] "Generic (PLEG): container finished" podID="575ac171-83b1-4d32-bc19-9a039f885c0a" containerID="feeeaf737ec76220a88d7b95c0b56dbf3b06b18d343cfa406fd59269a1adf883" exitCode=0 Nov 02 18:02:41.641679 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:41.641491 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" event={"ID":"575ac171-83b1-4d32-bc19-9a039f885c0a","Type":"ContainerDied","Data":"feeeaf737ec76220a88d7b95c0b56dbf3b06b18d343cfa406fd59269a1adf883"} Nov 02 18:02:41.642121 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:41.642104 5601 scope.go:122] "RemoveContainer" containerID="feeeaf737ec76220a88d7b95c0b56dbf3b06b18d343cfa406fd59269a1adf883" Nov 02 18:02:41.644230 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:41.644197 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:41.644325 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:41.644279 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:41.644325 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:41.644313 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:02:42.192764 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:42.192729 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj"] Nov 02 18:02:42.196274 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:42.196246 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" Nov 02 18:02:42.201939 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:42.201905 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-5c3c1f-serving-cert\"" Nov 02 18:02:42.216583 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:42.216539 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj"] Nov 02 18:02:42.357255 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:42.357216 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-metrics-certs\") pod \"maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj\" (UID: \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" Nov 02 18:02:42.357668 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:42.357358 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw8w7\" (UniqueName: \"kubernetes.io/projected/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-kube-api-access-cw8w7\") pod \"maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj\" (UID: \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" Nov 02 18:02:42.357668 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:42.357405 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj\" (UID: \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" Nov 02 18:02:42.357668 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:42.357633 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj\" (UID: \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" Nov 02 18:02:42.461355 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:42.458891 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-metrics-certs\") pod \"maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj\" (UID: \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" Nov 02 18:02:42.461355 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:42.459006 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw8w7\" (UniqueName: \"kubernetes.io/projected/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-kube-api-access-cw8w7\") pod \"maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj\" (UID: \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" Nov 02 18:02:42.461355 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:42.459035 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj\" (UID: \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" Nov 02 18:02:42.461355 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:42.459210 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj\" (UID: \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" Nov 02 18:02:42.461355 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:42.461325 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj\" (UID: \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" Nov 02 18:02:42.465554 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:42.465168 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-metrics-certs\") pod \"maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj\" (UID: \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" Nov 02 18:02:42.465860 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:42.465602 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj\" (UID: \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" Nov 02 18:02:42.469065 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:42.469042 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw8w7\" (UniqueName: \"kubernetes.io/projected/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-kube-api-access-cw8w7\") pod \"maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj\" (UID: \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" Nov 02 18:02:42.538810 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:42.538767 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" Nov 02 18:02:42.677350 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:42.675281 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" event={"ID":"575ac171-83b1-4d32-bc19-9a039f885c0a","Type":"ContainerStarted","Data":"f71922eb9dcdf9562dce8036c729ff9ba3fc3d85fb32951c45edacf28b8ddb8f"} Nov 02 18:02:42.898753 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:42.898715 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj"] Nov 02 18:02:42.916653 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:02:42.901274 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e6a7d0_45c4_4fce_aad6_edf7d92c2795.slice/crio-7c212f720c99b545759f7a57bea0268097a1980a7a40d2d34b5490ef1af6858b WatchSource:0}: Error finding container 7c212f720c99b545759f7a57bea0268097a1980a7a40d2d34b5490ef1af6858b: Status 404 returned error can't find the container with id 7c212f720c99b545759f7a57bea0268097a1980a7a40d2d34b5490ef1af6858b Nov 02 18:02:43.514022 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:43.513979 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:02:43.516392 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:43.516365 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 18:02:43.516513 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:43.516426 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" podUID="575ac171-83b1-4d32-bc19-9a039f885c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 18:02:43.727634 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:43.727527 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" event={"ID":"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795","Type":"ContainerStarted","Data":"08d24af05b9a7153a6245b16f456439eaa533589531027780d8e50e380ae8206"} Nov 02 18:02:43.727901 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:43.727875 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" event={"ID":"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795","Type":"ContainerStarted","Data":"7c212f720c99b545759f7a57bea0268097a1980a7a40d2d34b5490ef1af6858b"} Nov 02 18:02:43.728048 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:43.728034 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" Nov 02 18:02:43.790312 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:43.790240 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" podStartSLOduration=1.228168815 podStartE2EDuration="1.790221148s" totalImagesPullingTime="562.052333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:43.788438104 +0000 UTC m=+1728.479492118" watchObservedRunningTime="2026-11-02 18:02:43.790221148 +0000 UTC m=+1728.481275162" Nov 02 18:02:44.513673 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:44.513636 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 18:02:44.513854 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:44.513690 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" podUID="575ac171-83b1-4d32-bc19-9a039f885c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 18:02:45.514414 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:45.514373 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 18:02:45.514873 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:45.514435 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" podUID="575ac171-83b1-4d32-bc19-9a039f885c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 18:02:46.514237 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:46.514205 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 18:02:46.514423 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:46.514260 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" podUID="575ac171-83b1-4d32-bc19-9a039f885c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 18:02:47.521645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:47.516830 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 18:02:47.521645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:47.516894 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" podUID="575ac171-83b1-4d32-bc19-9a039f885c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 18:02:48.519208 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:48.514755 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 18:02:48.519208 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:48.514849 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" podUID="575ac171-83b1-4d32-bc19-9a039f885c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 18:02:49.518857 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:49.518826 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:02:49.519370 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:49.519263 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:02:49.520095 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:49.520073 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:02:49.820932 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:49.820834 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" Nov 02 18:02:53.902226 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:53.902179 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm"] Nov 02 18:02:53.907136 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:53.907110 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" Nov 02 18:02:53.918991 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:53.918578 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-bee8f6-serving-cert\"" Nov 02 18:02:53.919141 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:53.919018 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm"] Nov 02 18:02:54.068020 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:54.067351 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145f6bf1-b596-4e1e-a561-10c46c7c9b63-metrics-certs\") pod \"maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm\" (UID: \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" Nov 02 18:02:54.068020 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:54.067547 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdsq4\" (UniqueName: \"kubernetes.io/projected/145f6bf1-b596-4e1e-a561-10c46c7c9b63-kube-api-access-hdsq4\") pod \"maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm\" (UID: \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" Nov 02 18:02:54.068020 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:54.067692 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145f6bf1-b596-4e1e-a561-10c46c7c9b63-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm\" (UID: \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" Nov 02 18:02:54.068020 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:54.067819 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145f6bf1-b596-4e1e-a561-10c46c7c9b63-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm\" (UID: \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" Nov 02 18:02:54.169755 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:54.169731 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdsq4\" (UniqueName: \"kubernetes.io/projected/145f6bf1-b596-4e1e-a561-10c46c7c9b63-kube-api-access-hdsq4\") pod \"maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm\" (UID: \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" Nov 02 18:02:54.169914 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:54.169837 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145f6bf1-b596-4e1e-a561-10c46c7c9b63-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm\" (UID: \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" Nov 02 18:02:54.169961 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:54.169933 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145f6bf1-b596-4e1e-a561-10c46c7c9b63-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm\" (UID: \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" Nov 02 18:02:54.170084 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:54.170068 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145f6bf1-b596-4e1e-a561-10c46c7c9b63-metrics-certs\") pod \"maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm\" (UID: \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" Nov 02 18:02:54.170517 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:54.170495 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145f6bf1-b596-4e1e-a561-10c46c7c9b63-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm\" (UID: \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" Nov 02 18:02:54.172407 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:54.172382 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145f6bf1-b596-4e1e-a561-10c46c7c9b63-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm\" (UID: \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" Nov 02 18:02:54.172498 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:54.172432 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145f6bf1-b596-4e1e-a561-10c46c7c9b63-metrics-certs\") pod \"maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm\" (UID: \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" Nov 02 18:02:54.181186 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:54.181160 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdsq4\" (UniqueName: \"kubernetes.io/projected/145f6bf1-b596-4e1e-a561-10c46c7c9b63-kube-api-access-hdsq4\") pod \"maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm\" (UID: \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" Nov 02 18:02:54.234017 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:54.233984 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" Nov 02 18:02:54.520217 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:02:54.520170 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145f6bf1_b596_4e1e_a561_10c46c7c9b63.slice/crio-ba48037596af70caf7908e736f5054b5219fe0d09581d62a98c5ab1f9f8d18e2 WatchSource:0}: Error finding container ba48037596af70caf7908e736f5054b5219fe0d09581d62a98c5ab1f9f8d18e2: Status 404 returned error can't find the container with id ba48037596af70caf7908e736f5054b5219fe0d09581d62a98c5ab1f9f8d18e2 Nov 02 18:02:54.521433 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:54.521408 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm"] Nov 02 18:02:55.103403 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:55.103359 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" event={"ID":"145f6bf1-b596-4e1e-a561-10c46c7c9b63","Type":"ContainerStarted","Data":"f7be535fd735982751bec3196dfe6fa59fa2a5973661aab9ef4987587b2ae5e6"} Nov 02 18:02:55.103995 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:55.103429 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" Nov 02 18:02:55.103995 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:55.103452 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" event={"ID":"145f6bf1-b596-4e1e-a561-10c46c7c9b63","Type":"ContainerStarted","Data":"ba48037596af70caf7908e736f5054b5219fe0d09581d62a98c5ab1f9f8d18e2"} Nov 02 18:02:55.125005 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:02:55.124948 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" podStartSLOduration=1.732831373 podStartE2EDuration="2.124933423s" totalImagesPullingTime="392.10205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:55.123814979 +0000 UTC m=+1739.814868991" watchObservedRunningTime="2026-11-02 18:02:55.124933423 +0000 UTC m=+1739.815987435" Nov 02 18:03:01.149069 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:01.148818 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" Nov 02 18:03:04.618400 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:03:04.618330 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:03:05.894494 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:05.894458 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv"] Nov 02 18:03:05.932935 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:05.927936 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:05.933171 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:05.927830 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml"] Nov 02 18:03:05.987708 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:05.987677 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv"] Nov 02 18:03:05.988734 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:05.987809 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:05.989379 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:05.987960 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml"] Nov 02 18:03:05.989530 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:05.989516 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b"] Nov 02 18:03:05.989840 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:05.989814 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" podUID="09ee14b3-20d9-424b-beb3-211cb4ad7ae1" containerName="istio-proxy" containerID="cri-o://a8ffe84470fc33a5e11fc5acc0df077fe90bd345fe542f3e7e22c7cd3259e9f0" gracePeriod=30 Nov 02 18:03:06.020642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.018594 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17487351-07f6-4585-aa0c-80766361b815-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.020642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.018701 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.020642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.018737 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.020642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.018774 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.020642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.018844 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.020642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.018873 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.020642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.018896 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbh6f\" (UniqueName: \"kubernetes.io/projected/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-kube-api-access-vbh6f\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.020642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.018949 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.020642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.018982 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.020642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.019028 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhdnf\" (UniqueName: \"kubernetes.io/projected/17487351-07f6-4585-aa0c-80766361b815-kube-api-access-qhdnf\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.141643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.133797 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17487351-07f6-4585-aa0c-80766361b815-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.141643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.133886 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17487351-07f6-4585-aa0c-80766361b815-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.141643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.134017 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.141643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.134341 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.141643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.134385 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.141643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.134418 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.141643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.134478 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.141643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.134545 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.204781 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.168956 5601 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:03:06.204781 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.169053 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" podUID="09ee14b3-20d9-424b-beb3-211cb4ad7ae1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:03:06.262674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.244649 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.262674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.244716 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.262674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.244756 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbh6f\" (UniqueName: \"kubernetes.io/projected/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-kube-api-access-vbh6f\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.262674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.244820 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.262674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.244862 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.262674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.244915 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhdnf\" (UniqueName: \"kubernetes.io/projected/17487351-07f6-4585-aa0c-80766361b815-kube-api-access-qhdnf\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.262674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.244983 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17487351-07f6-4585-aa0c-80766361b815-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.262674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.245037 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17487351-07f6-4585-aa0c-80766361b815-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.262674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.245142 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.262674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.245259 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.262674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.245304 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.262674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.245337 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.262674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.245398 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.262674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.245469 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.262674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.245534 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17487351-07f6-4585-aa0c-80766361b815-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.262674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.245605 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.262674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.245660 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.262674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.245695 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.262674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.253863 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.289752 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.265189 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.289752 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.265548 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.289752 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.266222 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.289752 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.267847 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.289752 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.268137 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.289752 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.275099 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.289752 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.275888 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.289752 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.276552 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.314643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.309155 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17487351-07f6-4585-aa0c-80766361b815-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.344120 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.343897 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhdnf\" (UniqueName: \"kubernetes.io/projected/17487351-07f6-4585-aa0c-80766361b815-kube-api-access-qhdnf\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.365460 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.364230 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbh6f\" (UniqueName: \"kubernetes.io/projected/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-kube-api-access-vbh6f\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.365460 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.364682 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.365460 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.364946 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17487351-07f6-4585-aa0c-80766361b815-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.365460 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.365153 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.366669 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.366642 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.366963 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.366938 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.368866 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.368842 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17487351-07f6-4585-aa0c-80766361b815-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.580857 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.580778 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:06.609162 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.609074 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:06.929219 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.929184 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv"] Nov 02 18:03:06.931208 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:03:06.931181 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfef12c4_5397_4b54_8d3a_cb4ad6ec8c7a.slice/crio-680edc99eb18c2a5117e0e07400686b7ec585151e46e8dc850299b38a280f754 WatchSource:0}: Error finding container 680edc99eb18c2a5117e0e07400686b7ec585151e46e8dc850299b38a280f754: Status 404 returned error can't find the container with id 680edc99eb18c2a5117e0e07400686b7ec585151e46e8dc850299b38a280f754 Nov 02 18:03:06.934668 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.934631 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:03:06.934783 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.934715 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:03:06.934783 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.934765 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:03:06.988034 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.988007 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml"] Nov 02 18:03:06.994070 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:03:06.994047 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17487351_07f6_4585_aa0c_80766361b815.slice/crio-9a073a25fbbeb9fd2d8995a30b3a275241b3a71fa981df55cd440eaf580c1ba0 WatchSource:0}: Error finding container 9a073a25fbbeb9fd2d8995a30b3a275241b3a71fa981df55cd440eaf580c1ba0: Status 404 returned error can't find the container with id 9a073a25fbbeb9fd2d8995a30b3a275241b3a71fa981df55cd440eaf580c1ba0 Nov 02 18:03:06.997142 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.997107 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:03:06.997210 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.997179 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:03:06.997247 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:06.997208 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:03:07.361741 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:07.361649 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" event={"ID":"17487351-07f6-4585-aa0c-80766361b815","Type":"ContainerStarted","Data":"e9c373df4090677e381f08144a011f7b05d4224e756b54d22fa52275cae1f35b"} Nov 02 18:03:07.361741 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:07.361720 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" event={"ID":"17487351-07f6-4585-aa0c-80766361b815","Type":"ContainerStarted","Data":"9a073a25fbbeb9fd2d8995a30b3a275241b3a71fa981df55cd440eaf580c1ba0"} Nov 02 18:03:07.363939 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:07.363897 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" event={"ID":"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a","Type":"ContainerStarted","Data":"97f547e8a4ae6c6549a9de00ac8b2b2d6067394f0337c3688a5f9a663c0676a0"} Nov 02 18:03:07.364095 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:07.363958 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" event={"ID":"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a","Type":"ContainerStarted","Data":"680edc99eb18c2a5117e0e07400686b7ec585151e46e8dc850299b38a280f754"} Nov 02 18:03:07.385744 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:07.385680 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" podStartSLOduration=2.385666221 podStartE2EDuration="2.385666221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:07.383320128 +0000 UTC m=+1752.074374141" watchObservedRunningTime="2026-11-02 18:03:07.385666221 +0000 UTC m=+1752.076720233" Nov 02 18:03:07.423426 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:07.423352 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" podStartSLOduration=2.423332262 podStartE2EDuration="2.423332262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:07.418155054 +0000 UTC m=+1752.109209066" watchObservedRunningTime="2026-11-02 18:03:07.423332262 +0000 UTC m=+1752.114386277" Nov 02 18:03:07.586671 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:07.581286 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:07.651695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:07.637818 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:08.603646 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:08.599741 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 18:03:08.603646 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:08.599825 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" podUID="cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 18:03:08.623247 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:08.622785 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 18:03:08.623247 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:08.622848 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" podUID="17487351-07f6-4585-aa0c-80766361b815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 18:03:09.587091 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:09.587042 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 18:03:09.587406 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:09.587374 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" podUID="cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 18:03:09.621198 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:09.621154 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 18:03:09.621684 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:09.621236 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" podUID="17487351-07f6-4585-aa0c-80766361b815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 18:03:10.587640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:10.582004 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 18:03:10.587640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:10.582066 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" podUID="cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 18:03:10.599646 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:10.594726 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:10.619639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:10.610147 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:10.619639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:10.615975 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 18:03:10.619639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:10.616025 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" podUID="17487351-07f6-4585-aa0c-80766361b815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 18:03:10.636642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:10.635705 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:10.636642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:10.636099 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:10.797516 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:10.796769 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:11.162643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:11.160713 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:11.821655 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:11.819978 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 18:03:11.821655 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:11.820054 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" podUID="cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 18:03:11.966331 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:11.962263 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:11.966331 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:11.963048 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 18:03:12.136423 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:12.136301 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 18:03:12.146923 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:12.139564 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" podUID="17487351-07f6-4585-aa0c-80766361b815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 18:03:12.355212 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:12.342330 5601 generic.go:356] "Generic (PLEG): container finished" podID="09ee14b3-20d9-424b-beb3-211cb4ad7ae1" containerID="a8ffe84470fc33a5e11fc5acc0df077fe90bd345fe542f3e7e22c7cd3259e9f0" exitCode=0 Nov 02 18:03:12.355212 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:12.342594 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" event={"ID":"09ee14b3-20d9-424b-beb3-211cb4ad7ae1","Type":"ContainerDied","Data":"a8ffe84470fc33a5e11fc5acc0df077fe90bd345fe542f3e7e22c7cd3259e9f0"} Nov 02 18:03:12.479916 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:12.479858 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:12.588812 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:12.583168 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 18:03:12.588812 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:12.583232 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" podUID="cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 18:03:12.623820 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:12.622742 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 18:03:12.623820 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:12.622804 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" podUID="17487351-07f6-4585-aa0c-80766361b815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 18:03:12.832545 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:12.829741 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:12.919978 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:12.912603 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 18:03:12.990575 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:12.990536 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:03:13.342645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.336798 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-workload-certs\") pod \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " Nov 02 18:03:13.342645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.336899 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-kube-api-access-ws5r4\" (UniqueName: \"kubernetes.io/projected/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-kube-api-access-ws5r4\") pod \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " Nov 02 18:03:13.342645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.337039 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-token\" (UniqueName: \"kubernetes.io/projected/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-token\") pod \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " Nov 02 18:03:13.342645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.337081 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-envoy\") pod \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " Nov 02 18:03:13.342645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.337126 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-podinfo\") pod \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " Nov 02 18:03:13.342645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.337254 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-workload-socket\") pod \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " Nov 02 18:03:13.342645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.337338 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istiod-ca-cert\") pod \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " Nov 02 18:03:13.342645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.337414 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-credential-socket\") pod \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " Nov 02 18:03:13.342645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.337565 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-data\") pod \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\" (UID: \"09ee14b3-20d9-424b-beb3-211cb4ad7ae1\") " Nov 02 18:03:13.342645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.338701 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-data" pod "09ee14b3-20d9-424b-beb3-211cb4ad7ae1" (UID: "09ee14b3-20d9-424b-beb3-211cb4ad7ae1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:13.343416 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.343007 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-workload-certs" pod "09ee14b3-20d9-424b-beb3-211cb4ad7ae1" (UID: "09ee14b3-20d9-424b-beb3-211cb4ad7ae1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:13.439751 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.371089 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-workload-socket" pod "09ee14b3-20d9-424b-beb3-211cb4ad7ae1" (UID: "09ee14b3-20d9-424b-beb3-211cb4ad7ae1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:13.439751 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.371454 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istiod-ca-cert" pod "09ee14b3-20d9-424b-beb3-211cb4ad7ae1" (UID: "09ee14b3-20d9-424b-beb3-211cb4ad7ae1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:03:13.439751 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.371799 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-credential-socket" pod "09ee14b3-20d9-424b-beb3-211cb4ad7ae1" (UID: "09ee14b3-20d9-424b-beb3-211cb4ad7ae1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:13.461690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.458701 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:13.461690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.458737 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:13.461690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.458753 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:13.461690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.458766 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:13.461690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.458781 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:13.623782 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.620287 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 18:03:13.623782 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.620358 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" podUID="cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 18:03:13.636348 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.636305 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 18:03:13.637349 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.637311 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" podUID="17487351-07f6-4585-aa0c-80766361b815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 18:03:13.741441 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.738811 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7"] Nov 02 18:03:13.741441 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.739316 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-kube-api-access-ws5r4" pod "09ee14b3-20d9-424b-beb3-211cb4ad7ae1" (UID: "09ee14b3-20d9-424b-beb3-211cb4ad7ae1"). InnerVolumeSpecName "kube-api-access-ws5r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:13.741441 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.740174 5601 cpu_manager.go:414] "containerMap: removing container" podUID="09ee14b3-20d9-424b-beb3-211cb4ad7ae1" containerName="istio-proxy" Nov 02 18:03:13.741441 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.740167 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-token" pod "09ee14b3-20d9-424b-beb3-211cb4ad7ae1" (UID: "09ee14b3-20d9-424b-beb3-211cb4ad7ae1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:13.741441 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.740197 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ee14b3-20d9-424b-beb3-211cb4ad7ae1" containerName="istio-proxy" Nov 02 18:03:13.754648 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.753130 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-envoy" pod "09ee14b3-20d9-424b-beb3-211cb4ad7ae1" (UID: "09ee14b3-20d9-424b-beb3-211cb4ad7ae1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:13.754961 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.754934 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" Nov 02 18:03:13.787677 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.771009 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ee14b3-20d9-424b-beb3-211cb4ad7ae1" containerName="istio-proxy" Nov 02 18:03:13.787677 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.782492 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7"] Nov 02 18:03:13.787677 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.782538 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b" event={"ID":"09ee14b3-20d9-424b-beb3-211cb4ad7ae1","Type":"ContainerDied","Data":"8383fd0c9f8a22aa3721976562b067dfbaf4b25529499f6966f95562bdaca643"} Nov 02 18:03:13.787677 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.782605 5601 scope.go:122] "RemoveContainer" containerID="a8ffe84470fc33a5e11fc5acc0df077fe90bd345fe542f3e7e22c7cd3259e9f0" Nov 02 18:03:13.787677 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.782927 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" Nov 02 18:03:13.792929 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.792895 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5c3c1f\"/\"payload-processing-external-model-e2e--d55ca30b-dockercfg-tzv4c\"" Nov 02 18:03:13.831736 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.796122 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c3c1f\"/\"kube-root-ca.crt\"" Nov 02 18:03:13.831928 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.796199 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5c3c1f\"/\"agc-cred-6ae4ad932d9b\"" Nov 02 18:03:13.832038 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.796562 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c3c1f\"/\"routing-overlay\"" Nov 02 18:03:13.832144 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.796689 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c3c1f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5c3c1f\"" Nov 02 18:03:13.835637 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.796723 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c3c1f\"/\"openshift-service-ca.crt\"" Nov 02 18:03:13.837381 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.812191 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-podinfo" pod "09ee14b3-20d9-424b-beb3-211cb4ad7ae1" (UID: "09ee14b3-20d9-424b-beb3-211cb4ad7ae1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:03:13.873582 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.873536 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7\" (UID: \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\") " pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" Nov 02 18:03:13.901642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.901591 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpq5g\" (UniqueName: \"kubernetes.io/projected/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-kube-api-access-gpq5g\") pod \"payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7\" (UID: \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\") " pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" Nov 02 18:03:13.902032 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.902003 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7\" (UID: \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\") " pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" Nov 02 18:03:13.902149 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.902057 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7\" (UID: \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\") " pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" Nov 02 18:03:13.902324 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.902306 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:13.902407 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.902331 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:13.902407 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.902347 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:13.902407 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:13.902363 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws5r4\" (UniqueName: \"kubernetes.io/projected/09ee14b3-20d9-424b-beb3-211cb4ad7ae1-kube-api-access-ws5r4\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:14.003837 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:14.003785 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpq5g\" (UniqueName: \"kubernetes.io/projected/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-kube-api-access-gpq5g\") pod \"payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7\" (UID: \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\") " pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" Nov 02 18:03:14.004010 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:14.003895 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7\" (UID: \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\") " pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" Nov 02 18:03:14.004010 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:14.003919 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7\" (UID: \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\") " pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" Nov 02 18:03:14.004085 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:14.004062 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7\" (UID: \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\") " pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" Nov 02 18:03:14.005431 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:14.005403 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7\" (UID: \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\") " pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" Nov 02 18:03:14.006239 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:14.006217 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7\" (UID: \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\") " pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" Nov 02 18:03:14.007999 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:14.007979 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7\" (UID: \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\") " pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" Nov 02 18:03:14.018016 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:14.017985 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpq5g\" (UniqueName: \"kubernetes.io/projected/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-kube-api-access-gpq5g\") pod \"payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7\" (UID: \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\") " pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" Nov 02 18:03:14.125970 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:14.125930 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b"] Nov 02 18:03:14.206685 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:14.194843 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-h5r7b"] Nov 02 18:03:14.225867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:14.225827 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" Nov 02 18:03:14.600520 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:14.594051 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 18:03:14.600520 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:14.594110 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" podUID="cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 18:03:14.615087 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:14.615054 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 18:03:14.615237 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:14.615102 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" podUID="17487351-07f6-4585-aa0c-80766361b815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 18:03:14.693829 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:14.693797 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7"] Nov 02 18:03:14.836923 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:14.836823 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" event={"ID":"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d","Type":"ContainerStarted","Data":"d909d35e5b6cbf4cc5875683e8465c748fd34895d3d03d627ec10e740d551bf5"} Nov 02 18:03:15.437641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:15.434181 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ee14b3-20d9-424b-beb3-211cb4ad7ae1" path="/var/lib/kubelet/pods/09ee14b3-20d9-424b-beb3-211cb4ad7ae1/volumes" Nov 02 18:03:15.585640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:15.584725 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 18:03:15.585640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:15.584789 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" podUID="cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 18:03:15.615345 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:15.615085 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 18:03:15.615345 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:15.615130 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" podUID="17487351-07f6-4585-aa0c-80766361b815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 18:03:15.931642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:15.926864 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" event={"ID":"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d","Type":"ContainerStarted","Data":"957e6e1e4280251fde62bf55f3829950bac903812d7a9d47849a55ad28dd0995"} Nov 02 18:03:15.931642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:15.928220 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" Nov 02 18:03:15.986640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:15.966902 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" podStartSLOduration=3.966881882 podStartE2EDuration="3.966881882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:15.963592706 +0000 UTC m=+1760.654646724" watchObservedRunningTime="2026-11-02 18:03:15.966881882 +0000 UTC m=+1760.657935894" Nov 02 18:03:16.588784 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:16.588743 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:16.589451 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:16.589422 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:16.590313 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:16.590294 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:03:16.619999 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:16.619972 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:16.620349 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:16.620327 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:16.621260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:16.621233 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:03:27.955638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:27.954123 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" Nov 02 18:03:29.840716 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:29.840677 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2"] Nov 02 18:03:29.848281 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:29.846757 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" Nov 02 18:03:29.854336 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:29.854298 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"agc-cred-5041e53fa043\"" Nov 02 18:03:29.854775 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:29.854750 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"kube-root-ca.crt\"" Nov 02 18:03:29.855021 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:29.854999 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"payload-processing-external-model-plugins-e2e-extmod-b-bee8f6\"" Nov 02 18:03:29.855279 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:29.855261 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"routing-overlay\"" Nov 02 18:03:29.855403 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:29.855301 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"openshift-service-ca.crt\"" Nov 02 18:03:29.855513 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:29.855486 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"payload-processing-external-model-e2e--24159a0a-dockercfg-ndqxg\"" Nov 02 18:03:29.867526 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:29.867492 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2"] Nov 02 18:03:29.971213 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:29.970940 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2\" (UID: \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" Nov 02 18:03:29.971213 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:29.971060 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2\" (UID: \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" Nov 02 18:03:29.971213 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:29.971211 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxn8n\" (UniqueName: \"kubernetes.io/projected/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-kube-api-access-vxn8n\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2\" (UID: \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" Nov 02 18:03:29.971534 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:29.971311 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2\" (UID: \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" Nov 02 18:03:30.135729 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:30.133715 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2\" (UID: \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" Nov 02 18:03:30.135729 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:30.133843 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2\" (UID: \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" Nov 02 18:03:30.135729 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:30.134014 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxn8n\" (UniqueName: \"kubernetes.io/projected/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-kube-api-access-vxn8n\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2\" (UID: \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" Nov 02 18:03:30.135729 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:30.134130 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2\" (UID: \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" Nov 02 18:03:30.135729 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:30.135050 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2\" (UID: \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" Nov 02 18:03:30.137648 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:30.136304 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2\" (UID: \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" Nov 02 18:03:30.159517 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:30.159388 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2\" (UID: \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" Nov 02 18:03:30.161527 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:30.161484 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxn8n\" (UniqueName: \"kubernetes.io/projected/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-kube-api-access-vxn8n\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2\" (UID: \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" Nov 02 18:03:30.170413 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:30.170368 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" Nov 02 18:03:30.493334 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:30.493300 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2"] Nov 02 18:03:30.495099 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:03:30.495071 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf78776_4a73_4aa0_9e93_f3e5c4890bcc.slice/crio-1b11a3bcd26c97f47e88a0fec5c41950988406181aa3878783fad51ced145f3c WatchSource:0}: Error finding container 1b11a3bcd26c97f47e88a0fec5c41950988406181aa3878783fad51ced145f3c: Status 404 returned error can't find the container with id 1b11a3bcd26c97f47e88a0fec5c41950988406181aa3878783fad51ced145f3c Nov 02 18:03:31.167033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:31.166995 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" event={"ID":"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc","Type":"ContainerStarted","Data":"11df4a4a16ef3536fd83b3bedcd40b080540a4aed0508424866425f6840a92ed"} Nov 02 18:03:31.167451 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:31.167068 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" Nov 02 18:03:31.167451 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:31.167095 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" event={"ID":"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc","Type":"ContainerStarted","Data":"1b11a3bcd26c97f47e88a0fec5c41950988406181aa3878783fad51ced145f3c"} Nov 02 18:03:31.186954 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:31.186906 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" podStartSLOduration=2.186891229 podStartE2EDuration="2.186891229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:31.183950854 +0000 UTC m=+1775.875004867" watchObservedRunningTime="2026-11-02 18:03:31.186891229 +0000 UTC m=+1775.877945243" Nov 02 18:03:34.486378 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:03:34.486338 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:03:42.179488 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:03:42.179454 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" Nov 02 18:04:04.487937 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:04:04.487901 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:04:34.487654 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:04:34.487605 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:05:02.565082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:02.565044 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7"] Nov 02 18:05:04.487449 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:05:04.487408 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:05:07.566183 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.566125 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" podUID="0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d" containerName="payload-processing" containerID="cri-o://957e6e1e4280251fde62bf55f3829950bac903812d7a9d47849a55ad28dd0995" gracePeriod=25 Nov 02 18:05:07.865378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.865347 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" Nov 02 18:05:07.898156 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.898120 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w"] Nov 02 18:05:07.899047 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.899025 5601 cpu_manager.go:414] "containerMap: removing container" podUID="0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d" containerName="payload-processing" Nov 02 18:05:07.899047 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.899044 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d" containerName="payload-processing" Nov 02 18:05:07.899278 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.899175 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d" containerName="payload-processing" Nov 02 18:05:07.902827 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.902804 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" Nov 02 18:05:07.907780 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.907700 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"payload-processing-external-model-plugins-e2e-extmod-b-327608\"" Nov 02 18:05:07.907780 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.907742 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"agc-cred-6ce668d70ed3\"" Nov 02 18:05:07.907780 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.907754 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"openshift-service-ca.crt\"" Nov 02 18:05:07.908017 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.907787 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"agc-cred-d1a3311a2190\"" Nov 02 18:05:07.910447 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.910423 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"kube-root-ca.crt\"" Nov 02 18:05:07.910604 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.910491 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"payload-processing-external-model-e2e--69602eb2-dockercfg-ffvnd\"" Nov 02 18:05:07.915921 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.915896 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w"] Nov 02 18:05:07.916146 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.916123 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"routing-overlay\"" Nov 02 18:05:07.967231 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.967201 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-plugins-config-volume\") pod \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\" (UID: \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\") " Nov 02 18:05:07.967432 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.967240 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-kube-api-access-gpq5g\" (UniqueName: \"kubernetes.io/projected/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-kube-api-access-gpq5g\") pod \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\" (UID: \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\") " Nov 02 18:05:07.967515 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.967440 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-routing-overlay-volume\") pod \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\" (UID: \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\") " Nov 02 18:05:07.967670 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.967606 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-provider-credentials\") pod \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\" (UID: \"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d\") " Nov 02 18:05:07.967937 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.967914 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-plugins-config-volume" pod "0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d" (UID: "0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:07.968029 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.967929 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-routing-overlay-volume" pod "0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d" (UID: "0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:07.968086 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.968066 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/924f6e47-49a4-4084-a49a-e3717b2dad87-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-5c557f84w\" (UID: \"924f6e47-49a4-4084-a49a-e3717b2dad87\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" Nov 02 18:05:07.968794 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.968365 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-327608-5c557f84w\" (UID: \"924f6e47-49a4-4084-a49a-e3717b2dad87\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" Nov 02 18:05:07.968794 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.968413 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924f6e47-49a4-4084-a49a-e3717b2dad87-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-5c557f84w\" (UID: \"924f6e47-49a4-4084-a49a-e3717b2dad87\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" Nov 02 18:05:07.968794 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.968476 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2jrx\" (UniqueName: \"kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-kube-api-access-x2jrx\") pod \"payload-processing-external-model-e2e-extmod-b-327608-5c557f84w\" (UID: \"924f6e47-49a4-4084-a49a-e3717b2dad87\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" Nov 02 18:05:07.968794 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.968702 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:05:07.968794 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.968724 5601 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-routing-overlay-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:05:07.970188 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.970164 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-provider-credentials" pod "0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d" (UID: "0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:07.970294 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:07.970200 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-kube-api-access-gpq5g" pod "0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d" (UID: "0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d"). InnerVolumeSpecName "kube-api-access-gpq5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:08.070490 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.070453 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-327608-5c557f84w\" (UID: \"924f6e47-49a4-4084-a49a-e3717b2dad87\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" Nov 02 18:05:08.070697 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.070496 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924f6e47-49a4-4084-a49a-e3717b2dad87-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-5c557f84w\" (UID: \"924f6e47-49a4-4084-a49a-e3717b2dad87\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" Nov 02 18:05:08.070697 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.070540 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2jrx\" (UniqueName: \"kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-kube-api-access-x2jrx\") pod \"payload-processing-external-model-e2e-extmod-b-327608-5c557f84w\" (UID: \"924f6e47-49a4-4084-a49a-e3717b2dad87\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" Nov 02 18:05:08.070839 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.070715 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/924f6e47-49a4-4084-a49a-e3717b2dad87-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-5c557f84w\" (UID: \"924f6e47-49a4-4084-a49a-e3717b2dad87\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" Nov 02 18:05:08.070945 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.070925 5601 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-provider-credentials\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:05:08.071012 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.070949 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpq5g\" (UniqueName: \"kubernetes.io/projected/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d-kube-api-access-gpq5g\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:05:08.071400 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.071374 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/924f6e47-49a4-4084-a49a-e3717b2dad87-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-5c557f84w\" (UID: \"924f6e47-49a4-4084-a49a-e3717b2dad87\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" Nov 02 18:05:08.071519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.071492 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924f6e47-49a4-4084-a49a-e3717b2dad87-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-5c557f84w\" (UID: \"924f6e47-49a4-4084-a49a-e3717b2dad87\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" Nov 02 18:05:08.073378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.073350 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-327608-5c557f84w\" (UID: \"924f6e47-49a4-4084-a49a-e3717b2dad87\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" Nov 02 18:05:08.080068 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.080011 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2jrx\" (UniqueName: \"kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-kube-api-access-x2jrx\") pod \"payload-processing-external-model-e2e-extmod-b-327608-5c557f84w\" (UID: \"924f6e47-49a4-4084-a49a-e3717b2dad87\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" Nov 02 18:05:08.225249 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.225213 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" Nov 02 18:05:08.454309 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.454259 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2"] Nov 02 18:05:08.571527 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.571487 5601 generic.go:356] "Generic (PLEG): container finished" podID="0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d" containerID="957e6e1e4280251fde62bf55f3829950bac903812d7a9d47849a55ad28dd0995" exitCode=0 Nov 02 18:05:08.571982 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.571690 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" event={"ID":"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d","Type":"ContainerDied","Data":"957e6e1e4280251fde62bf55f3829950bac903812d7a9d47849a55ad28dd0995"} Nov 02 18:05:08.571982 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.571750 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" event={"ID":"0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d","Type":"ContainerDied","Data":"d909d35e5b6cbf4cc5875683e8465c748fd34895d3d03d627ec10e740d551bf5"} Nov 02 18:05:08.571982 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.571824 5601 scope.go:122] "RemoveContainer" containerID="957e6e1e4280251fde62bf55f3829950bac903812d7a9d47849a55ad28dd0995" Nov 02 18:05:08.572087 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.572001 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7" Nov 02 18:05:08.580745 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.580719 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w"] Nov 02 18:05:08.581128 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:05:08.581101 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924f6e47_49a4_4084_a49a_e3717b2dad87.slice/crio-05b8d8d8807c4471732fcde593c01ea28a6af7e0907f1bde4f64a4b72d791ec9 WatchSource:0}: Error finding container 05b8d8d8807c4471732fcde593c01ea28a6af7e0907f1bde4f64a4b72d791ec9: Status 404 returned error can't find the container with id 05b8d8d8807c4471732fcde593c01ea28a6af7e0907f1bde4f64a4b72d791ec9 Nov 02 18:05:08.587314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.587292 5601 scope.go:122] "RemoveContainer" containerID="957e6e1e4280251fde62bf55f3829950bac903812d7a9d47849a55ad28dd0995" Nov 02 18:05:08.587601 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:05:08.587565 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"957e6e1e4280251fde62bf55f3829950bac903812d7a9d47849a55ad28dd0995\": container with ID starting with 957e6e1e4280251fde62bf55f3829950bac903812d7a9d47849a55ad28dd0995 not found: ID does not exist" containerID="957e6e1e4280251fde62bf55f3829950bac903812d7a9d47849a55ad28dd0995" Nov 02 18:05:08.587694 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.587606 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"957e6e1e4280251fde62bf55f3829950bac903812d7a9d47849a55ad28dd0995"} err="failed to get container status \"957e6e1e4280251fde62bf55f3829950bac903812d7a9d47849a55ad28dd0995\": rpc error: code = NotFound desc = could not find container \"957e6e1e4280251fde62bf55f3829950bac903812d7a9d47849a55ad28dd0995\": container with ID starting with 957e6e1e4280251fde62bf55f3829950bac903812d7a9d47849a55ad28dd0995 not found: ID does not exist" Nov 02 18:05:08.706640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.703700 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7"] Nov 02 18:05:08.736823 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:08.736050 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5c3c1f/payload-processing-external-model-e2e-extmod-b-5c3c1f-8c7bp2mx7"] Nov 02 18:05:09.428816 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:09.428779 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d" path="/var/lib/kubelet/pods/0e20c3fe-ebc0-4a9d-a74e-449ca7a3f85d/volumes" Nov 02 18:05:09.581728 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:09.581689 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" event={"ID":"924f6e47-49a4-4084-a49a-e3717b2dad87","Type":"ContainerStarted","Data":"d923d52b4d79814f26a17522ea42e228d3fed1bd579783ead76a3da2543ee82d"} Nov 02 18:05:09.582179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:09.581747 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" event={"ID":"924f6e47-49a4-4084-a49a-e3717b2dad87","Type":"ContainerStarted","Data":"05b8d8d8807c4471732fcde593c01ea28a6af7e0907f1bde4f64a4b72d791ec9"} Nov 02 18:05:09.582179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:09.581786 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" Nov 02 18:05:09.608894 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:09.608835 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" podStartSLOduration=45.608821394 podStartE2EDuration="45.608821394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:09.606921402 +0000 UTC m=+1874.297975410" watchObservedRunningTime="2026-11-02 18:05:09.608821394 +0000 UTC m=+1874.299875406" Nov 02 18:05:13.456446 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.456398 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" podUID="0cf78776-4a73-4aa0-9e93-f3e5c4890bcc" containerName="payload-processing" containerID="cri-o://11df4a4a16ef3536fd83b3bedcd40b080540a4aed0508424866425f6840a92ed" gracePeriod=25 Nov 02 18:05:13.619342 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.619307 5601 generic.go:356] "Generic (PLEG): container finished" podID="0cf78776-4a73-4aa0-9e93-f3e5c4890bcc" containerID="11df4a4a16ef3536fd83b3bedcd40b080540a4aed0508424866425f6840a92ed" exitCode=0 Nov 02 18:05:13.619558 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.619388 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" event={"ID":"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc","Type":"ContainerDied","Data":"11df4a4a16ef3536fd83b3bedcd40b080540a4aed0508424866425f6840a92ed"} Nov 02 18:05:13.811005 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.810975 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" Nov 02 18:05:13.845439 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.845403 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr"] Nov 02 18:05:13.846207 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.846185 5601 cpu_manager.go:414] "containerMap: removing container" podUID="0cf78776-4a73-4aa0-9e93-f3e5c4890bcc" containerName="payload-processing" Nov 02 18:05:13.846207 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.846204 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cf78776-4a73-4aa0-9e93-f3e5c4890bcc" containerName="payload-processing" Nov 02 18:05:13.846391 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.846349 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cf78776-4a73-4aa0-9e93-f3e5c4890bcc" containerName="payload-processing" Nov 02 18:05:13.850266 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.850243 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" Nov 02 18:05:13.855999 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.855967 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ef96f\"/\"agc-cred-871c0b31c1f7\"" Nov 02 18:05:13.856164 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.856094 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ef96f\"/\"kube-root-ca.crt\"" Nov 02 18:05:13.856231 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.856205 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ef96f\"/\"routing-overlay\"" Nov 02 18:05:13.856287 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.856269 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ef96f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1ef96f\"" Nov 02 18:05:13.856344 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.856325 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ef96f\"/\"openshift-service-ca.crt\"" Nov 02 18:05:13.856467 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.856449 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ef96f\"/\"payload-processing-external-model-e2e--72426c61-dockercfg-2zfsk\"" Nov 02 18:05:13.856543 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.856530 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ef96f\"/\"agc-cred-2f1dcc18e33e\"" Nov 02 18:05:13.859200 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.859173 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr"] Nov 02 18:05:13.919492 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.919457 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-kube-api-access-vxn8n\" (UniqueName: \"kubernetes.io/projected/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-kube-api-access-vxn8n\") pod \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\" (UID: \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\") " Nov 02 18:05:13.919688 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.919510 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-provider-credentials\") pod \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\" (UID: \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\") " Nov 02 18:05:13.919688 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.919561 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-plugins-config-volume\") pod \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\" (UID: \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\") " Nov 02 18:05:13.919796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.919711 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-routing-overlay-volume\") pod \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\" (UID: \"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc\") " Nov 02 18:05:13.920123 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.920086 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-routing-overlay-volume" pod "0cf78776-4a73-4aa0-9e93-f3e5c4890bcc" (UID: "0cf78776-4a73-4aa0-9e93-f3e5c4890bcc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:13.920230 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.920129 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-plugins-config-volume" pod "0cf78776-4a73-4aa0-9e93-f3e5c4890bcc" (UID: "0cf78776-4a73-4aa0-9e93-f3e5c4890bcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:13.920230 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.920214 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr\" (UID: \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" Nov 02 18:05:13.920493 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.920472 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr\" (UID: \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" Nov 02 18:05:13.920563 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.920533 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr\" (UID: \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" Nov 02 18:05:13.920682 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.920666 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2qq8\" (UniqueName: \"kubernetes.io/projected/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-kube-api-access-b2qq8\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr\" (UID: \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" Nov 02 18:05:13.920820 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.920807 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:05:13.920881 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.920822 5601 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-routing-overlay-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:05:13.921944 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.921921 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-provider-credentials" pod "0cf78776-4a73-4aa0-9e93-f3e5c4890bcc" (UID: "0cf78776-4a73-4aa0-9e93-f3e5c4890bcc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:13.922070 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:13.922044 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-kube-api-access-vxn8n" pod "0cf78776-4a73-4aa0-9e93-f3e5c4890bcc" (UID: "0cf78776-4a73-4aa0-9e93-f3e5c4890bcc"). InnerVolumeSpecName "kube-api-access-vxn8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:14.032695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.029216 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr\" (UID: \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" Nov 02 18:05:14.032695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.029320 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr\" (UID: \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" Nov 02 18:05:14.032695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.029434 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2qq8\" (UniqueName: \"kubernetes.io/projected/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-kube-api-access-b2qq8\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr\" (UID: \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" Nov 02 18:05:14.032695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.029541 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr\" (UID: \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" Nov 02 18:05:14.032695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.029828 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxn8n\" (UniqueName: \"kubernetes.io/projected/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-kube-api-access-vxn8n\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:05:14.032695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.029856 5601 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc-provider-credentials\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:05:14.038093 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.035888 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr\" (UID: \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" Nov 02 18:05:14.038093 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.037135 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr\" (UID: \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" Nov 02 18:05:14.059239 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.048120 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr\" (UID: \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" Nov 02 18:05:14.059239 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.054306 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2qq8\" (UniqueName: \"kubernetes.io/projected/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-kube-api-access-b2qq8\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr\" (UID: \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" Nov 02 18:05:14.171179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.171145 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" Nov 02 18:05:14.460096 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:05:14.460054 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3127d21_f37a_4f37_9a41_87bd7f48ceb1.slice/crio-a397aa7c3de734d6752c004dbe1435ebb8b629005606aebd9d8da26eb58bac9a WatchSource:0}: Error finding container a397aa7c3de734d6752c004dbe1435ebb8b629005606aebd9d8da26eb58bac9a: Status 404 returned error can't find the container with id a397aa7c3de734d6752c004dbe1435ebb8b629005606aebd9d8da26eb58bac9a Nov 02 18:05:14.460823 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.460799 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr"] Nov 02 18:05:14.630903 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.630869 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" Nov 02 18:05:14.630903 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.630882 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2" event={"ID":"0cf78776-4a73-4aa0-9e93-f3e5c4890bcc","Type":"ContainerDied","Data":"1b11a3bcd26c97f47e88a0fec5c41950988406181aa3878783fad51ced145f3c"} Nov 02 18:05:14.631155 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.630949 5601 scope.go:122] "RemoveContainer" containerID="11df4a4a16ef3536fd83b3bedcd40b080540a4aed0508424866425f6840a92ed" Nov 02 18:05:14.645640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.638698 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" event={"ID":"c3127d21-f37a-4f37-9a41-87bd7f48ceb1","Type":"ContainerStarted","Data":"7dbdc8928fa17427305fd2faeef1318250b5cf33190cad583070abb874c8230f"} Nov 02 18:05:14.645640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.638765 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" event={"ID":"c3127d21-f37a-4f37-9a41-87bd7f48ceb1","Type":"ContainerStarted","Data":"a397aa7c3de734d6752c004dbe1435ebb8b629005606aebd9d8da26eb58bac9a"} Nov 02 18:05:14.645640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.639713 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" Nov 02 18:05:14.662602 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.662542 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" podStartSLOduration=49.662525741 podStartE2EDuration="49.662525741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:14.66009688 +0000 UTC m=+1879.351150895" watchObservedRunningTime="2026-11-02 18:05:14.662525741 +0000 UTC m=+1879.353579752" Nov 02 18:05:14.679000 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.678966 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2"] Nov 02 18:05:14.683263 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:14.683231 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-bd54zv4c2"] Nov 02 18:05:15.428955 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:15.428925 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cf78776-4a73-4aa0-9e93-f3e5c4890bcc" path="/var/lib/kubelet/pods/0cf78776-4a73-4aa0-9e93-f3e5c4890bcc/volumes" Nov 02 18:05:20.594846 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:20.594801 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" Nov 02 18:05:23.305388 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:23.305353 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr"] Nov 02 18:05:23.308362 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:23.308335 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" Nov 02 18:05:28.305952 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.305908 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" podUID="c3127d21-f37a-4f37-9a41-87bd7f48ceb1" containerName="payload-processing" containerID="cri-o://7dbdc8928fa17427305fd2faeef1318250b5cf33190cad583070abb874c8230f" gracePeriod=25 Nov 02 18:05:28.678989 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.678964 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" Nov 02 18:05:28.721006 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.720972 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm"] Nov 02 18:05:28.721176 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.721086 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-plugins-config-volume\") pod \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\" (UID: \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\") " Nov 02 18:05:28.721283 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.721264 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-provider-credentials\") pod \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\" (UID: \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\") " Nov 02 18:05:28.721424 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.721403 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-routing-overlay-volume\") pod \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\" (UID: \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\") " Nov 02 18:05:28.721543 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.721518 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-plugins-config-volume" pod "c3127d21-f37a-4f37-9a41-87bd7f48ceb1" (UID: "c3127d21-f37a-4f37-9a41-87bd7f48ceb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:28.721702 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.721678 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-kube-api-access-b2qq8\" (UniqueName: \"kubernetes.io/projected/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-kube-api-access-b2qq8\") pod \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\" (UID: \"c3127d21-f37a-4f37-9a41-87bd7f48ceb1\") " Nov 02 18:05:28.721916 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.721894 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-routing-overlay-volume" pod "c3127d21-f37a-4f37-9a41-87bd7f48ceb1" (UID: "c3127d21-f37a-4f37-9a41-87bd7f48ceb1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:28.722121 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.722100 5601 cpu_manager.go:414] "containerMap: removing container" podUID="c3127d21-f37a-4f37-9a41-87bd7f48ceb1" containerName="payload-processing" Nov 02 18:05:28.722190 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.722122 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3127d21-f37a-4f37-9a41-87bd7f48ceb1" containerName="payload-processing" Nov 02 18:05:28.722364 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.722351 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3127d21-f37a-4f37-9a41-87bd7f48ceb1" containerName="payload-processing" Nov 02 18:05:28.722667 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.722644 5601 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-routing-overlay-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:05:28.722760 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.722672 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:05:28.740536 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.724998 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-kube-api-access-b2qq8" pod "c3127d21-f37a-4f37-9a41-87bd7f48ceb1" (UID: "c3127d21-f37a-4f37-9a41-87bd7f48ceb1"). InnerVolumeSpecName "kube-api-access-b2qq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:28.740536 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.726085 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-provider-credentials" pod "c3127d21-f37a-4f37-9a41-87bd7f48ceb1" (UID: "c3127d21-f37a-4f37-9a41-87bd7f48ceb1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:28.754870 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.754848 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" Nov 02 18:05:28.756013 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.755988 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm"] Nov 02 18:05:28.759795 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.759772 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ef96f\"/\"agc-cred-4fb1f87404ff\"" Nov 02 18:05:28.833549 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.833507 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04201ee2-1052-4e30-a2f5-037dde6c08f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm\" (UID: \"04201ee2-1052-4e30-a2f5-037dde6c08f5\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" Nov 02 18:05:28.833746 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.833584 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm\" (UID: \"04201ee2-1052-4e30-a2f5-037dde6c08f5\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" Nov 02 18:05:28.833746 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.833653 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04201ee2-1052-4e30-a2f5-037dde6c08f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm\" (UID: \"04201ee2-1052-4e30-a2f5-037dde6c08f5\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" Nov 02 18:05:28.833934 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.833912 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7drmp\" (UniqueName: \"kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-kube-api-access-7drmp\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm\" (UID: \"04201ee2-1052-4e30-a2f5-037dde6c08f5\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" Nov 02 18:05:28.834052 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.834039 5601 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-provider-credentials\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:05:28.834096 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.834056 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2qq8\" (UniqueName: \"kubernetes.io/projected/c3127d21-f37a-4f37-9a41-87bd7f48ceb1-kube-api-access-b2qq8\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:05:28.935844 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.935810 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7drmp\" (UniqueName: \"kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-kube-api-access-7drmp\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm\" (UID: \"04201ee2-1052-4e30-a2f5-037dde6c08f5\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" Nov 02 18:05:28.936024 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.935927 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04201ee2-1052-4e30-a2f5-037dde6c08f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm\" (UID: \"04201ee2-1052-4e30-a2f5-037dde6c08f5\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" Nov 02 18:05:28.936024 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.935964 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm\" (UID: \"04201ee2-1052-4e30-a2f5-037dde6c08f5\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" Nov 02 18:05:28.936024 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.935990 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04201ee2-1052-4e30-a2f5-037dde6c08f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm\" (UID: \"04201ee2-1052-4e30-a2f5-037dde6c08f5\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" Nov 02 18:05:28.936690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.936665 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04201ee2-1052-4e30-a2f5-037dde6c08f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm\" (UID: \"04201ee2-1052-4e30-a2f5-037dde6c08f5\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" Nov 02 18:05:28.936768 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.936710 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04201ee2-1052-4e30-a2f5-037dde6c08f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm\" (UID: \"04201ee2-1052-4e30-a2f5-037dde6c08f5\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" Nov 02 18:05:28.938575 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.938558 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm\" (UID: \"04201ee2-1052-4e30-a2f5-037dde6c08f5\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" Nov 02 18:05:28.945796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:28.945774 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7drmp\" (UniqueName: \"kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-kube-api-access-7drmp\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm\" (UID: \"04201ee2-1052-4e30-a2f5-037dde6c08f5\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" Nov 02 18:05:29.066793 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:29.066760 5601 generic.go:356] "Generic (PLEG): container finished" podID="c3127d21-f37a-4f37-9a41-87bd7f48ceb1" containerID="7dbdc8928fa17427305fd2faeef1318250b5cf33190cad583070abb874c8230f" exitCode=0 Nov 02 18:05:29.066980 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:29.066808 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" Nov 02 18:05:29.066980 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:29.066837 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" event={"ID":"c3127d21-f37a-4f37-9a41-87bd7f48ceb1","Type":"ContainerDied","Data":"7dbdc8928fa17427305fd2faeef1318250b5cf33190cad583070abb874c8230f"} Nov 02 18:05:29.066980 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:29.066890 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr" event={"ID":"c3127d21-f37a-4f37-9a41-87bd7f48ceb1","Type":"ContainerDied","Data":"a397aa7c3de734d6752c004dbe1435ebb8b629005606aebd9d8da26eb58bac9a"} Nov 02 18:05:29.066980 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:29.066916 5601 scope.go:122] "RemoveContainer" containerID="7dbdc8928fa17427305fd2faeef1318250b5cf33190cad583070abb874c8230f" Nov 02 18:05:29.078422 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:29.078400 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" Nov 02 18:05:29.079070 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:29.079053 5601 scope.go:122] "RemoveContainer" containerID="7dbdc8928fa17427305fd2faeef1318250b5cf33190cad583070abb874c8230f" Nov 02 18:05:29.079334 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:05:29.079315 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dbdc8928fa17427305fd2faeef1318250b5cf33190cad583070abb874c8230f\": container with ID starting with 7dbdc8928fa17427305fd2faeef1318250b5cf33190cad583070abb874c8230f not found: ID does not exist" containerID="7dbdc8928fa17427305fd2faeef1318250b5cf33190cad583070abb874c8230f" Nov 02 18:05:29.079376 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:29.079343 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dbdc8928fa17427305fd2faeef1318250b5cf33190cad583070abb874c8230f"} err="failed to get container status \"7dbdc8928fa17427305fd2faeef1318250b5cf33190cad583070abb874c8230f\": rpc error: code = NotFound desc = could not find container \"7dbdc8928fa17427305fd2faeef1318250b5cf33190cad583070abb874c8230f\": container with ID starting with 7dbdc8928fa17427305fd2faeef1318250b5cf33190cad583070abb874c8230f not found: ID does not exist" Nov 02 18:05:29.098518 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:29.098497 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr"] Nov 02 18:05:29.102725 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:29.102703 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-8567xrlzr"] Nov 02 18:05:29.367259 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:29.367230 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm"] Nov 02 18:05:29.367670 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:05:29.367324 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04201ee2_1052_4e30_a2f5_037dde6c08f5.slice/crio-f41f47cfb9e6ed2793c9b0ffcf25174a942c0636b91b9f52649e872cc1f33ff9 WatchSource:0}: Error finding container f41f47cfb9e6ed2793c9b0ffcf25174a942c0636b91b9f52649e872cc1f33ff9: Status 404 returned error can't find the container with id f41f47cfb9e6ed2793c9b0ffcf25174a942c0636b91b9f52649e872cc1f33ff9 Nov 02 18:05:29.456947 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:29.456852 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3127d21-f37a-4f37-9a41-87bd7f48ceb1" path="/var/lib/kubelet/pods/c3127d21-f37a-4f37-9a41-87bd7f48ceb1/volumes" Nov 02 18:05:30.079384 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:30.079349 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" event={"ID":"04201ee2-1052-4e30-a2f5-037dde6c08f5","Type":"ContainerStarted","Data":"78c7230918e71329b27692f92fb5d6a2c5779eb5e47cd716aac4b9a9fa8cb94a"} Nov 02 18:05:30.079562 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:30.079406 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" event={"ID":"04201ee2-1052-4e30-a2f5-037dde6c08f5","Type":"ContainerStarted","Data":"f41f47cfb9e6ed2793c9b0ffcf25174a942c0636b91b9f52649e872cc1f33ff9"} Nov 02 18:05:30.079562 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:30.079433 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" Nov 02 18:05:30.102896 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:30.102847 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" podStartSLOduration=7.102833739 podStartE2EDuration="7.102833739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:30.099141783 +0000 UTC m=+1894.790195795" watchObservedRunningTime="2026-11-02 18:05:30.102833739 +0000 UTC m=+1894.793887750" Nov 02 18:05:34.486225 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:05:34.486187 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:05:41.092157 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:05:41.092063 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" Nov 02 18:06:04.488090 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:04.488047 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:06:34.486957 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:34.486916 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:06:39.302184 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:39.302124 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm"] Nov 02 18:06:39.302851 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:39.302493 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" podUID="29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" containerName="istio-proxy" containerID="cri-o://ef9e829b4a0b0daafbf84838306e4bc44c37cde7a2f04a06de8872a552f5de88" gracePeriod=30 Nov 02 18:06:41.543301 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:41.543263 5601 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ef96f/agc-cred-2f1dcc18e33e: secret "agc-cred-2f1dcc18e33e" not found Nov 02 18:06:41.543301 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:41.543294 5601 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ef96f/agc-cred-4fb1f87404ff: secret "agc-cred-4fb1f87404ff" not found Nov 02 18:06:41.543301 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:41.543310 5601 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm: [secret "agc-cred-2f1dcc18e33e" not found, secret "agc-cred-4fb1f87404ff" not found] Nov 02 18:06:41.544005 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:41.543369 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-provider-credentials podName:04201ee2-1052-4e30-a2f5-037dde6c08f5 nodeName:}" failed. No retries permitted until 2026-11-02 18:06:42.043352318 +0000 UTC m=+1966.734406335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" (UID: "04201ee2-1052-4e30-a2f5-037dde6c08f5") : [secret "agc-cred-2f1dcc18e33e" not found, secret "agc-cred-4fb1f87404ff" not found] Nov 02 18:06:42.054049 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:42.054005 5601 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ef96f/agc-cred-2f1dcc18e33e: secret "agc-cred-2f1dcc18e33e" not found Nov 02 18:06:42.054049 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:42.054040 5601 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ef96f/agc-cred-4fb1f87404ff: secret "agc-cred-4fb1f87404ff" not found Nov 02 18:06:42.054049 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:42.054055 5601 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm: [secret "agc-cred-2f1dcc18e33e" not found, secret "agc-cred-4fb1f87404ff" not found] Nov 02 18:06:42.054377 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:42.054110 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-provider-credentials podName:04201ee2-1052-4e30-a2f5-037dde6c08f5 nodeName:}" failed. No retries permitted until 2026-11-02 18:06:43.05409546 +0000 UTC m=+1967.745149450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" (UID: "04201ee2-1052-4e30-a2f5-037dde6c08f5") : [secret "agc-cred-2f1dcc18e33e" not found, secret "agc-cred-4fb1f87404ff" not found] Nov 02 18:06:43.075044 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:43.075002 5601 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ef96f/agc-cred-2f1dcc18e33e: secret "agc-cred-2f1dcc18e33e" not found Nov 02 18:06:43.075636 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:43.075605 5601 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ef96f/agc-cred-4fb1f87404ff: secret "agc-cred-4fb1f87404ff" not found Nov 02 18:06:43.075774 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:43.075759 5601 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm: [secret "agc-cred-2f1dcc18e33e" not found, secret "agc-cred-4fb1f87404ff" not found] Nov 02 18:06:43.075955 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:43.075941 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-provider-credentials podName:04201ee2-1052-4e30-a2f5-037dde6c08f5 nodeName:}" failed. No retries permitted until 2026-11-02 18:06:45.075917194 +0000 UTC m=+1969.766971199 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" (UID: "04201ee2-1052-4e30-a2f5-037dde6c08f5") : [secret "agc-cred-2f1dcc18e33e" not found, secret "agc-cred-4fb1f87404ff" not found] Nov 02 18:06:44.626704 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.626669 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 18:06:44.673879 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.673841 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb"] Nov 02 18:06:44.674755 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.674732 5601 cpu_manager.go:414] "containerMap: removing container" podUID="29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" containerName="istio-proxy" Nov 02 18:06:44.674755 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.674751 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" containerName="istio-proxy" Nov 02 18:06:44.674941 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.674910 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" containerName="istio-proxy" Nov 02 18:06:44.678920 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.678867 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" Nov 02 18:06:44.688729 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.688597 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277"] Nov 02 18:06:44.700391 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.700338 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb"] Nov 02 18:06:44.700573 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.700526 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" Nov 02 18:06:44.703435 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.703402 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277"] Nov 02 18:06:44.708240 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.708209 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-327608\"/\"agc-cred-21e65f396d52\"" Nov 02 18:06:44.741926 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.741891 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istiod-ca-cert\") pod \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " Nov 02 18:06:44.742127 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.741965 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-kube-api-access-8pjdn\" (UniqueName: \"kubernetes.io/projected/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-kube-api-access-8pjdn\") pod \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " Nov 02 18:06:44.742127 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.742004 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-podinfo\") pod \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " Nov 02 18:06:44.742260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.742159 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-workload-certs\") pod \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " Nov 02 18:06:44.742260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.742191 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-envoy\") pod \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " Nov 02 18:06:44.742260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.742214 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istiod-ca-cert" pod "29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" (UID: "29f3bb67-cbce-4b09-b4fe-dccc94daf1bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:06:44.742260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.742231 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-data\") pod \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " Nov 02 18:06:44.742474 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.742269 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-credential-socket\") pod \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " Nov 02 18:06:44.742474 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.742294 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-workload-socket\") pod \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " Nov 02 18:06:44.742474 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.742459 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-token\" (UniqueName: \"kubernetes.io/projected/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-token\") pod \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\" (UID: \"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb\") " Nov 02 18:06:44.742679 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.742520 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-workload-certs" pod "29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" (UID: "29f3bb67-cbce-4b09-b4fe-dccc94daf1bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:06:44.742756 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.742739 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-data" pod "29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" (UID: "29f3bb67-cbce-4b09-b4fe-dccc94daf1bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:06:44.742878 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.742851 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bx5p\" (UniqueName: \"kubernetes.io/projected/1e43007a-4111-4b61-9497-268081a88b5b-kube-api-access-7bx5p\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb\" (UID: \"1e43007a-4111-4b61-9497-268081a88b5b\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" Nov 02 18:06:44.743709 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.742994 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e43007a-4111-4b61-9497-268081a88b5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb\" (UID: \"1e43007a-4111-4b61-9497-268081a88b5b\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" Nov 02 18:06:44.743709 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.743174 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e43007a-4111-4b61-9497-268081a88b5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb\" (UID: \"1e43007a-4111-4b61-9497-268081a88b5b\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" Nov 02 18:06:44.743709 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.743406 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e43007a-4111-4b61-9497-268081a88b5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb\" (UID: \"1e43007a-4111-4b61-9497-268081a88b5b\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" Nov 02 18:06:44.743709 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.743610 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:06:44.743709 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.743642 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-credential-socket" pod "29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" (UID: "29f3bb67-cbce-4b09-b4fe-dccc94daf1bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:06:44.743709 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.743649 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:06:44.744122 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.743723 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:06:44.744122 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.743854 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-workload-socket" pod "29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" (UID: "29f3bb67-cbce-4b09-b4fe-dccc94daf1bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:06:44.744961 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.744933 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-kube-api-access-8pjdn" pod "29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" (UID: "29f3bb67-cbce-4b09-b4fe-dccc94daf1bb"). InnerVolumeSpecName "kube-api-access-8pjdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:06:44.745349 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.745324 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-podinfo" pod "29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" (UID: "29f3bb67-cbce-4b09-b4fe-dccc94daf1bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:06:44.745499 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.745469 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-envoy" pod "29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" (UID: "29f3bb67-cbce-4b09-b4fe-dccc94daf1bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:06:44.745914 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.745882 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-token" pod "29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" (UID: "29f3bb67-cbce-4b09-b4fe-dccc94daf1bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:06:44.845042 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.844942 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e43007a-4111-4b61-9497-268081a88b5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb\" (UID: \"1e43007a-4111-4b61-9497-268081a88b5b\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" Nov 02 18:06:44.845243 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.845081 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-327608-7df9lw277\" (UID: \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" Nov 02 18:06:44.845243 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.845130 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e43007a-4111-4b61-9497-268081a88b5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb\" (UID: \"1e43007a-4111-4b61-9497-268081a88b5b\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" Nov 02 18:06:44.845367 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.845289 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-7df9lw277\" (UID: \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" Nov 02 18:06:44.845367 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.845349 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-7df9lw277\" (UID: \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" Nov 02 18:06:44.845476 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.845423 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwh9c\" (UniqueName: \"kubernetes.io/projected/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-kube-api-access-dwh9c\") pod \"payload-processing-external-model-e2e-extmod-b-327608-7df9lw277\" (UID: \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" Nov 02 18:06:44.845567 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.845543 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e43007a-4111-4b61-9497-268081a88b5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb\" (UID: \"1e43007a-4111-4b61-9497-268081a88b5b\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" Nov 02 18:06:44.845777 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.845761 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bx5p\" (UniqueName: \"kubernetes.io/projected/1e43007a-4111-4b61-9497-268081a88b5b-kube-api-access-7bx5p\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb\" (UID: \"1e43007a-4111-4b61-9497-268081a88b5b\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" Nov 02 18:06:44.845912 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.845899 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:06:44.845974 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.845918 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:06:44.845974 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.845933 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:06:44.845974 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.845947 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:06:44.845974 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.845959 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pjdn\" (UniqueName: \"kubernetes.io/projected/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-kube-api-access-8pjdn\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:06:44.845974 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.845970 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:06:44.846185 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.845973 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e43007a-4111-4b61-9497-268081a88b5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb\" (UID: \"1e43007a-4111-4b61-9497-268081a88b5b\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" Nov 02 18:06:44.846185 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.846170 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e43007a-4111-4b61-9497-268081a88b5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb\" (UID: \"1e43007a-4111-4b61-9497-268081a88b5b\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" Nov 02 18:06:44.847724 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.847704 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e43007a-4111-4b61-9497-268081a88b5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb\" (UID: \"1e43007a-4111-4b61-9497-268081a88b5b\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" Nov 02 18:06:44.854846 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.854821 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bx5p\" (UniqueName: \"kubernetes.io/projected/1e43007a-4111-4b61-9497-268081a88b5b-kube-api-access-7bx5p\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb\" (UID: \"1e43007a-4111-4b61-9497-268081a88b5b\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" Nov 02 18:06:44.930342 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.930301 5601 generic.go:356] "Generic (PLEG): container finished" podID="29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" containerID="ef9e829b4a0b0daafbf84838306e4bc44c37cde7a2f04a06de8872a552f5de88" exitCode=0 Nov 02 18:06:44.930553 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.930359 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" event={"ID":"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb","Type":"ContainerDied","Data":"ef9e829b4a0b0daafbf84838306e4bc44c37cde7a2f04a06de8872a552f5de88"} Nov 02 18:06:44.930553 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.930372 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" Nov 02 18:06:44.930553 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.930405 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm" event={"ID":"29f3bb67-cbce-4b09-b4fe-dccc94daf1bb","Type":"ContainerDied","Data":"12138ded2aae95778cde2b27e8d22bae69709ceb9ad514ca7cfdd33a258a37c7"} Nov 02 18:06:44.930553 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.930428 5601 scope.go:122] "RemoveContainer" containerID="ef9e829b4a0b0daafbf84838306e4bc44c37cde7a2f04a06de8872a552f5de88" Nov 02 18:06:44.947347 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.947312 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-327608-7df9lw277\" (UID: \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" Nov 02 18:06:44.947565 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.947431 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-7df9lw277\" (UID: \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" Nov 02 18:06:44.947565 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.947480 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-7df9lw277\" (UID: \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" Nov 02 18:06:44.947565 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.947512 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwh9c\" (UniqueName: \"kubernetes.io/projected/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-kube-api-access-dwh9c\") pod \"payload-processing-external-model-e2e-extmod-b-327608-7df9lw277\" (UID: \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" Nov 02 18:06:44.948875 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.948848 5601 scope.go:122] "RemoveContainer" containerID="ef9e829b4a0b0daafbf84838306e4bc44c37cde7a2f04a06de8872a552f5de88" Nov 02 18:06:44.950160 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:44.949783 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9e829b4a0b0daafbf84838306e4bc44c37cde7a2f04a06de8872a552f5de88\": container with ID starting with ef9e829b4a0b0daafbf84838306e4bc44c37cde7a2f04a06de8872a552f5de88 not found: ID does not exist" containerID="ef9e829b4a0b0daafbf84838306e4bc44c37cde7a2f04a06de8872a552f5de88" Nov 02 18:06:44.950160 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.949823 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9e829b4a0b0daafbf84838306e4bc44c37cde7a2f04a06de8872a552f5de88"} err="failed to get container status \"ef9e829b4a0b0daafbf84838306e4bc44c37cde7a2f04a06de8872a552f5de88\": rpc error: code = NotFound desc = could not find container \"ef9e829b4a0b0daafbf84838306e4bc44c37cde7a2f04a06de8872a552f5de88\": container with ID starting with ef9e829b4a0b0daafbf84838306e4bc44c37cde7a2f04a06de8872a552f5de88 not found: ID does not exist" Nov 02 18:06:44.950399 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.950358 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-7df9lw277\" (UID: \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" Nov 02 18:06:44.950957 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.950931 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-7df9lw277\" (UID: \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" Nov 02 18:06:44.951535 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.951511 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-327608-7df9lw277\" (UID: \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" Nov 02 18:06:44.958011 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.957982 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwh9c\" (UniqueName: \"kubernetes.io/projected/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-kube-api-access-dwh9c\") pod \"payload-processing-external-model-e2e-extmod-b-327608-7df9lw277\" (UID: \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" Nov 02 18:06:44.971062 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.970999 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm"] Nov 02 18:06:44.978038 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:44.978004 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-4wnqm"] Nov 02 18:06:45.005946 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:45.005904 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" Nov 02 18:06:45.018970 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:45.018942 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" Nov 02 18:06:45.152278 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:45.152204 5601 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ef96f/agc-cred-2f1dcc18e33e: secret "agc-cred-2f1dcc18e33e" not found Nov 02 18:06:45.152278 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:45.152233 5601 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ef96f/agc-cred-4fb1f87404ff: secret "agc-cred-4fb1f87404ff" not found Nov 02 18:06:45.152278 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:45.152247 5601 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm: [secret "agc-cred-2f1dcc18e33e" not found, secret "agc-cred-4fb1f87404ff" not found] Nov 02 18:06:45.152524 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:45.152295 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-provider-credentials podName:04201ee2-1052-4e30-a2f5-037dde6c08f5 nodeName:}" failed. No retries permitted until 2026-11-02 18:06:49.152281565 +0000 UTC m=+1973.843335554 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" (UID: "04201ee2-1052-4e30-a2f5-037dde6c08f5") : [secret "agc-cred-2f1dcc18e33e" not found, secret "agc-cred-4fb1f87404ff" not found] Nov 02 18:06:45.311847 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:45.311808 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb"] Nov 02 18:06:45.313122 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:06:45.312781 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e43007a_4111_4b61_9497_268081a88b5b.slice/crio-4602ed6e2ac3badb1862a4337cd117533203c98a3bb64b7151ad352afbfcf0f9 WatchSource:0}: Error finding container 4602ed6e2ac3badb1862a4337cd117533203c98a3bb64b7151ad352afbfcf0f9: Status 404 returned error can't find the container with id 4602ed6e2ac3badb1862a4337cd117533203c98a3bb64b7151ad352afbfcf0f9 Nov 02 18:06:45.346977 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:06:45.346948 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8e6bb8_a079_40d4_adc5_bb59e53d8d3f.slice/crio-4dcba857fbccfe38ec14d89634a0f4263727204509ef38caf8636313d852c0be WatchSource:0}: Error finding container 4dcba857fbccfe38ec14d89634a0f4263727204509ef38caf8636313d852c0be: Status 404 returned error can't find the container with id 4dcba857fbccfe38ec14d89634a0f4263727204509ef38caf8636313d852c0be Nov 02 18:06:45.351089 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:45.351065 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277"] Nov 02 18:06:45.438868 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:45.438836 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29f3bb67-cbce-4b09-b4fe-dccc94daf1bb" path="/var/lib/kubelet/pods/29f3bb67-cbce-4b09-b4fe-dccc94daf1bb/volumes" Nov 02 18:06:45.940543 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:45.940500 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" event={"ID":"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f","Type":"ContainerStarted","Data":"4a289a50fc404437c4ae349b9bdc3fd2e1aa8d27ee73603b80143bd03cbdd6f8"} Nov 02 18:06:45.941066 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:45.940593 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" event={"ID":"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f","Type":"ContainerStarted","Data":"4dcba857fbccfe38ec14d89634a0f4263727204509ef38caf8636313d852c0be"} Nov 02 18:06:45.941066 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:45.940658 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" Nov 02 18:06:45.942312 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:45.942284 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" event={"ID":"1e43007a-4111-4b61-9497-268081a88b5b","Type":"ContainerStarted","Data":"a0341775f81f6a5ba385df239f1a22dc1744a96926c982afd720768ac8bfbaed"} Nov 02 18:06:45.942455 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:45.942334 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" event={"ID":"1e43007a-4111-4b61-9497-268081a88b5b","Type":"ContainerStarted","Data":"4602ed6e2ac3badb1862a4337cd117533203c98a3bb64b7151ad352afbfcf0f9"} Nov 02 18:06:45.942455 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:45.942362 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" Nov 02 18:06:45.962262 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:45.962207 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" podStartSLOduration=55.962190235 podStartE2EDuration="55.962190235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:45.959143923 +0000 UTC m=+1970.650197935" watchObservedRunningTime="2026-11-02 18:06:45.962190235 +0000 UTC m=+1970.653244273" Nov 02 18:06:45.980311 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:45.980250 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" podStartSLOduration=28.980233835 podStartE2EDuration="28.980233835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:45.977394305 +0000 UTC m=+1970.668448331" watchObservedRunningTime="2026-11-02 18:06:45.980233835 +0000 UTC m=+1970.671287847" Nov 02 18:06:47.946749 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:47.946695 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277"] Nov 02 18:06:48.531409 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:48.531373 5601 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-327608/agc-cred-6ce668d70ed3: secret "agc-cred-6ce668d70ed3" not found Nov 02 18:06:48.531696 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:48.531676 5601 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w: secret "agc-cred-6ce668d70ed3" not found Nov 02 18:06:48.531902 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:48.531888 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-provider-credentials podName:924f6e47-49a4-4084-a49a-e3717b2dad87 nodeName:}" failed. No retries permitted until 2026-11-02 18:06:49.03186818 +0000 UTC m=+1973.722922186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" (UID: "924f6e47-49a4-4084-a49a-e3717b2dad87") : secret "agc-cred-6ce668d70ed3" not found Nov 02 18:06:49.044691 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:49.044655 5601 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-327608/agc-cred-6ce668d70ed3: secret "agc-cred-6ce668d70ed3" not found Nov 02 18:06:49.044691 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:49.044696 5601 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w: secret "agc-cred-6ce668d70ed3" not found Nov 02 18:06:49.045177 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:49.044773 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-provider-credentials podName:924f6e47-49a4-4084-a49a-e3717b2dad87 nodeName:}" failed. No retries permitted until 2026-11-02 18:06:50.044756517 +0000 UTC m=+1974.735810526 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" (UID: "924f6e47-49a4-4084-a49a-e3717b2dad87") : secret "agc-cred-6ce668d70ed3" not found Nov 02 18:06:49.248722 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:49.248690 5601 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ef96f/agc-cred-2f1dcc18e33e: secret "agc-cred-2f1dcc18e33e" not found Nov 02 18:06:49.248722 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:49.248716 5601 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ef96f/agc-cred-4fb1f87404ff: secret "agc-cred-4fb1f87404ff" not found Nov 02 18:06:49.248722 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:49.248728 5601 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm: [secret "agc-cred-2f1dcc18e33e" not found, secret "agc-cred-4fb1f87404ff" not found] Nov 02 18:06:49.248958 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:49.248792 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-provider-credentials podName:04201ee2-1052-4e30-a2f5-037dde6c08f5 nodeName:}" failed. No retries permitted until 2026-11-02 18:06:57.24877829 +0000 UTC m=+1981.939832297 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" (UID: "04201ee2-1052-4e30-a2f5-037dde6c08f5") : [secret "agc-cred-2f1dcc18e33e" not found, secret "agc-cred-4fb1f87404ff" not found] Nov 02 18:06:50.062176 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:50.062144 5601 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-327608/agc-cred-6ce668d70ed3: secret "agc-cred-6ce668d70ed3" not found Nov 02 18:06:50.062176 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:50.062179 5601 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w: secret "agc-cred-6ce668d70ed3" not found Nov 02 18:06:50.062743 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:50.062248 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-provider-credentials podName:924f6e47-49a4-4084-a49a-e3717b2dad87 nodeName:}" failed. No retries permitted until 2026-11-02 18:06:52.062227026 +0000 UTC m=+1976.753281016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" (UID: "924f6e47-49a4-4084-a49a-e3717b2dad87") : secret "agc-cred-6ce668d70ed3" not found Nov 02 18:06:52.107969 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:52.107931 5601 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-327608/agc-cred-6ce668d70ed3: secret "agc-cred-6ce668d70ed3" not found Nov 02 18:06:52.107969 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:52.107965 5601 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w: secret "agc-cred-6ce668d70ed3" not found Nov 02 18:06:52.108443 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:52.108019 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-provider-credentials podName:924f6e47-49a4-4084-a49a-e3717b2dad87 nodeName:}" failed. No retries permitted until 2026-11-02 18:06:56.108004334 +0000 UTC m=+1980.799058324 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" (UID: "924f6e47-49a4-4084-a49a-e3717b2dad87") : secret "agc-cred-6ce668d70ed3" not found Nov 02 18:06:52.966507 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:52.966454 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" podUID="6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f" containerName="payload-processing" containerID="cri-o://4a289a50fc404437c4ae349b9bdc3fd2e1aa8d27ee73603b80143bd03cbdd6f8" gracePeriod=25 Nov 02 18:06:53.249302 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.249274 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" Nov 02 18:06:53.286000 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.285951 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb"] Nov 02 18:06:53.287273 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.287246 5601 cpu_manager.go:414] "containerMap: removing container" podUID="6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f" containerName="payload-processing" Nov 02 18:06:53.287273 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.287275 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f" containerName="payload-processing" Nov 02 18:06:53.287486 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.287472 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f" containerName="payload-processing" Nov 02 18:06:53.291101 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.291075 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" Nov 02 18:06:53.298436 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.298409 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb"] Nov 02 18:06:53.334462 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.334427 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-plugins-config-volume\") pod \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\" (UID: \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\") " Nov 02 18:06:53.334973 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.334531 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-routing-overlay-volume\") pod \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\" (UID: \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\") " Nov 02 18:06:53.334973 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.334570 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-kube-api-access-dwh9c\" (UniqueName: \"kubernetes.io/projected/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-kube-api-access-dwh9c\") pod \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\" (UID: \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\") " Nov 02 18:06:53.334973 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.334858 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-provider-credentials\") pod \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\" (UID: \"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f\") " Nov 02 18:06:53.335211 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.335187 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e802fac6-939b-45d5-8bf7-1070a9aa256f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb\" (UID: \"e802fac6-939b-45d5-8bf7-1070a9aa256f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" Nov 02 18:06:53.335298 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.335283 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e802fac6-939b-45d5-8bf7-1070a9aa256f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb\" (UID: \"e802fac6-939b-45d5-8bf7-1070a9aa256f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" Nov 02 18:06:53.335393 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.335380 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw6wc\" (UniqueName: \"kubernetes.io/projected/e802fac6-939b-45d5-8bf7-1070a9aa256f-kube-api-access-mw6wc\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb\" (UID: \"e802fac6-939b-45d5-8bf7-1070a9aa256f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" Nov 02 18:06:53.335658 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.335602 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e802fac6-939b-45d5-8bf7-1070a9aa256f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb\" (UID: \"e802fac6-939b-45d5-8bf7-1070a9aa256f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" Nov 02 18:06:53.336128 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.336098 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-plugins-config-volume" pod "6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f" (UID: "6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:06:53.336419 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.336399 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-routing-overlay-volume" pod "6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f" (UID: "6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:06:53.343509 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.343333 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-kube-api-access-dwh9c" pod "6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f" (UID: "6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f"). InnerVolumeSpecName "kube-api-access-dwh9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:06:53.352213 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.352168 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-provider-credentials" pod "6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f" (UID: "6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:06:53.437451 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.437411 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e802fac6-939b-45d5-8bf7-1070a9aa256f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb\" (UID: \"e802fac6-939b-45d5-8bf7-1070a9aa256f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" Nov 02 18:06:53.437743 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.437498 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw6wc\" (UniqueName: \"kubernetes.io/projected/e802fac6-939b-45d5-8bf7-1070a9aa256f-kube-api-access-mw6wc\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb\" (UID: \"e802fac6-939b-45d5-8bf7-1070a9aa256f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" Nov 02 18:06:53.437743 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.437699 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e802fac6-939b-45d5-8bf7-1070a9aa256f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb\" (UID: \"e802fac6-939b-45d5-8bf7-1070a9aa256f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" Nov 02 18:06:53.437894 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.437789 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e802fac6-939b-45d5-8bf7-1070a9aa256f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb\" (UID: \"e802fac6-939b-45d5-8bf7-1070a9aa256f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" Nov 02 18:06:53.437894 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.437859 5601 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-provider-credentials\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:06:53.437894 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.437869 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:06:53.437894 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.437880 5601 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-routing-overlay-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:06:53.437894 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.437889 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwh9c\" (UniqueName: \"kubernetes.io/projected/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f-kube-api-access-dwh9c\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:06:53.440733 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.438718 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e802fac6-939b-45d5-8bf7-1070a9aa256f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb\" (UID: \"e802fac6-939b-45d5-8bf7-1070a9aa256f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" Nov 02 18:06:53.440733 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.440170 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e802fac6-939b-45d5-8bf7-1070a9aa256f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb\" (UID: \"e802fac6-939b-45d5-8bf7-1070a9aa256f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" Nov 02 18:06:53.441928 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.441905 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e802fac6-939b-45d5-8bf7-1070a9aa256f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb\" (UID: \"e802fac6-939b-45d5-8bf7-1070a9aa256f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" Nov 02 18:06:53.448645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.448598 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw6wc\" (UniqueName: \"kubernetes.io/projected/e802fac6-939b-45d5-8bf7-1070a9aa256f-kube-api-access-mw6wc\") pod \"payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb\" (UID: \"e802fac6-939b-45d5-8bf7-1070a9aa256f\") " pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" Nov 02 18:06:53.612337 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.612250 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" Nov 02 18:06:53.901151 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:53.901114 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb"] Nov 02 18:06:53.901319 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:06:53.901292 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode802fac6_939b_45d5_8bf7_1070a9aa256f.slice/crio-106eca8ea58a93c1a1635056cd2d12f0a9ceea917568b4b4656c70ca00daf84a WatchSource:0}: Error finding container 106eca8ea58a93c1a1635056cd2d12f0a9ceea917568b4b4656c70ca00daf84a: Status 404 returned error can't find the container with id 106eca8ea58a93c1a1635056cd2d12f0a9ceea917568b4b4656c70ca00daf84a Nov 02 18:06:54.017476 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:54.017437 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" event={"ID":"e802fac6-939b-45d5-8bf7-1070a9aa256f","Type":"ContainerStarted","Data":"c8ab79e9d239efcb04ce9957f0de38ed9f074beec65ccb6e219a57504e74af2b"} Nov 02 18:06:54.017714 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:54.017506 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" event={"ID":"e802fac6-939b-45d5-8bf7-1070a9aa256f","Type":"ContainerStarted","Data":"106eca8ea58a93c1a1635056cd2d12f0a9ceea917568b4b4656c70ca00daf84a"} Nov 02 18:06:54.017714 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:54.017663 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" Nov 02 18:06:54.019035 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:54.019009 5601 generic.go:356] "Generic (PLEG): container finished" podID="6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f" containerID="4a289a50fc404437c4ae349b9bdc3fd2e1aa8d27ee73603b80143bd03cbdd6f8" exitCode=0 Nov 02 18:06:54.019144 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:54.019050 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" Nov 02 18:06:54.019144 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:54.019097 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" event={"ID":"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f","Type":"ContainerDied","Data":"4a289a50fc404437c4ae349b9bdc3fd2e1aa8d27ee73603b80143bd03cbdd6f8"} Nov 02 18:06:54.019144 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:54.019138 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277" event={"ID":"6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f","Type":"ContainerDied","Data":"4dcba857fbccfe38ec14d89634a0f4263727204509ef38caf8636313d852c0be"} Nov 02 18:06:54.019259 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:54.019169 5601 scope.go:122] "RemoveContainer" containerID="4a289a50fc404437c4ae349b9bdc3fd2e1aa8d27ee73603b80143bd03cbdd6f8" Nov 02 18:06:54.033900 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:54.033877 5601 scope.go:122] "RemoveContainer" containerID="4a289a50fc404437c4ae349b9bdc3fd2e1aa8d27ee73603b80143bd03cbdd6f8" Nov 02 18:06:54.034216 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:54.034191 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a289a50fc404437c4ae349b9bdc3fd2e1aa8d27ee73603b80143bd03cbdd6f8\": container with ID starting with 4a289a50fc404437c4ae349b9bdc3fd2e1aa8d27ee73603b80143bd03cbdd6f8 not found: ID does not exist" containerID="4a289a50fc404437c4ae349b9bdc3fd2e1aa8d27ee73603b80143bd03cbdd6f8" Nov 02 18:06:54.034282 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:54.034224 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a289a50fc404437c4ae349b9bdc3fd2e1aa8d27ee73603b80143bd03cbdd6f8"} err="failed to get container status \"4a289a50fc404437c4ae349b9bdc3fd2e1aa8d27ee73603b80143bd03cbdd6f8\": rpc error: code = NotFound desc = could not find container \"4a289a50fc404437c4ae349b9bdc3fd2e1aa8d27ee73603b80143bd03cbdd6f8\": container with ID starting with 4a289a50fc404437c4ae349b9bdc3fd2e1aa8d27ee73603b80143bd03cbdd6f8 not found: ID does not exist" Nov 02 18:06:54.040831 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:54.040784 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" podStartSLOduration=7.040771492 podStartE2EDuration="7.040771492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:54.036608189 +0000 UTC m=+1978.727662204" watchObservedRunningTime="2026-11-02 18:06:54.040771492 +0000 UTC m=+1978.731825503" Nov 02 18:06:54.055893 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:54.055863 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277"] Nov 02 18:06:54.062334 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:54.062306 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-7df9lw277"] Nov 02 18:06:55.428412 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:55.428377 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f" path="/var/lib/kubelet/pods/6b8e6bb8-a079-40d4-adc5-bb59e53d8d3f/volumes" Nov 02 18:06:56.126734 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:56.126663 5601 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-327608/agc-cred-6ce668d70ed3: secret "agc-cred-6ce668d70ed3" not found Nov 02 18:06:56.126734 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:56.126700 5601 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w: secret "agc-cred-6ce668d70ed3" not found Nov 02 18:06:56.127008 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:56.126775 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-provider-credentials podName:924f6e47-49a4-4084-a49a-e3717b2dad87 nodeName:}" failed. No retries permitted until 2026-11-02 18:07:04.126758627 +0000 UTC m=+1988.817812637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" (UID: "924f6e47-49a4-4084-a49a-e3717b2dad87") : secret "agc-cred-6ce668d70ed3" not found Nov 02 18:06:56.956335 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:56.956303 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" Nov 02 18:06:57.013790 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:57.013741 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm"] Nov 02 18:06:57.350298 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:57.350212 5601 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ef96f/agc-cred-2f1dcc18e33e: secret "agc-cred-2f1dcc18e33e" not found Nov 02 18:06:57.350298 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:57.350240 5601 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ef96f/agc-cred-4fb1f87404ff: secret "agc-cred-4fb1f87404ff" not found Nov 02 18:06:57.350298 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:57.350254 5601 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm: [secret "agc-cred-2f1dcc18e33e" not found, secret "agc-cred-4fb1f87404ff" not found] Nov 02 18:06:57.350553 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:06:57.350317 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-provider-credentials podName:04201ee2-1052-4e30-a2f5-037dde6c08f5 nodeName:}" failed. No retries permitted until 2026-11-02 18:07:13.350301765 +0000 UTC m=+1998.041355759 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" (UID: "04201ee2-1052-4e30-a2f5-037dde6c08f5") : [secret "agc-cred-2f1dcc18e33e" not found, secret "agc-cred-4fb1f87404ff" not found] Nov 02 18:06:59.877089 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:06:59.877037 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb"] Nov 02 18:07:02.019765 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.019503 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" podUID="04201ee2-1052-4e30-a2f5-037dde6c08f5" containerName="payload-processing" containerID="cri-o://78c7230918e71329b27692f92fb5d6a2c5779eb5e47cd716aac4b9a9fa8cb94a" gracePeriod=25 Nov 02 18:07:02.234978 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.234936 5601 generic.go:356] "Generic (PLEG): container finished" podID="04201ee2-1052-4e30-a2f5-037dde6c08f5" containerID="78c7230918e71329b27692f92fb5d6a2c5779eb5e47cd716aac4b9a9fa8cb94a" exitCode=0 Nov 02 18:07:02.235268 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.235245 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" event={"ID":"04201ee2-1052-4e30-a2f5-037dde6c08f5","Type":"ContainerDied","Data":"78c7230918e71329b27692f92fb5d6a2c5779eb5e47cd716aac4b9a9fa8cb94a"} Nov 02 18:07:02.307831 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.307803 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" Nov 02 18:07:02.339085 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.339047 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8"] Nov 02 18:07:02.340457 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.340263 5601 cpu_manager.go:414] "containerMap: removing container" podUID="04201ee2-1052-4e30-a2f5-037dde6c08f5" containerName="payload-processing" Nov 02 18:07:02.340457 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.340288 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04201ee2-1052-4e30-a2f5-037dde6c08f5" containerName="payload-processing" Nov 02 18:07:02.340702 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.340498 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="04201ee2-1052-4e30-a2f5-037dde6c08f5" containerName="payload-processing" Nov 02 18:07:02.343516 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.343493 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" Nov 02 18:07:02.359683 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.359043 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"kube-root-ca.crt\"" Nov 02 18:07:02.359683 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.359144 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"payload-processing-external-model-plugins-e2e-extmod-a-807c08\"" Nov 02 18:07:02.359683 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.359218 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"payload-processing-external-model-e2e--d2178d1f-dockercfg-lmw2m\"" Nov 02 18:07:02.359683 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.359241 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"openshift-service-ca.crt\"" Nov 02 18:07:02.359683 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.359351 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"routing-overlay\"" Nov 02 18:07:02.359683 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.359456 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"agc-cred-a01632a9a9c9\"" Nov 02 18:07:02.369316 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.369286 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8"] Nov 02 18:07:02.389454 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.389416 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-provider-credentials\") pod \"04201ee2-1052-4e30-a2f5-037dde6c08f5\" (UID: \"04201ee2-1052-4e30-a2f5-037dde6c08f5\") " Nov 02 18:07:02.389637 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.389600 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-kube-api-access-7drmp\" (UniqueName: \"kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-kube-api-access-7drmp\") pod \"04201ee2-1052-4e30-a2f5-037dde6c08f5\" (UID: \"04201ee2-1052-4e30-a2f5-037dde6c08f5\") " Nov 02 18:07:02.389734 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.389719 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04201ee2-1052-4e30-a2f5-037dde6c08f5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04201ee2-1052-4e30-a2f5-037dde6c08f5-routing-overlay-volume\") pod \"04201ee2-1052-4e30-a2f5-037dde6c08f5\" (UID: \"04201ee2-1052-4e30-a2f5-037dde6c08f5\") " Nov 02 18:07:02.389909 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.389891 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04201ee2-1052-4e30-a2f5-037dde6c08f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04201ee2-1052-4e30-a2f5-037dde6c08f5-plugins-config-volume\") pod \"04201ee2-1052-4e30-a2f5-037dde6c08f5\" (UID: \"04201ee2-1052-4e30-a2f5-037dde6c08f5\") " Nov 02 18:07:02.390809 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.390163 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04201ee2-1052-4e30-a2f5-037dde6c08f5-routing-overlay-volume" pod "04201ee2-1052-4e30-a2f5-037dde6c08f5" (UID: "04201ee2-1052-4e30-a2f5-037dde6c08f5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:02.390809 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.390396 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04201ee2-1052-4e30-a2f5-037dde6c08f5-plugins-config-volume" pod "04201ee2-1052-4e30-a2f5-037dde6c08f5" (UID: "04201ee2-1052-4e30-a2f5-037dde6c08f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:02.390809 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.390760 5601 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04201ee2-1052-4e30-a2f5-037dde6c08f5-routing-overlay-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:02.390809 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.390782 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04201ee2-1052-4e30-a2f5-037dde6c08f5-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:02.391826 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.391801 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-kube-api-access-7drmp" pod "04201ee2-1052-4e30-a2f5-037dde6c08f5" (UID: "04201ee2-1052-4e30-a2f5-037dde6c08f5"). InnerVolumeSpecName "kube-api-access-7drmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:02.391919 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.391836 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-provider-credentials" pod "04201ee2-1052-4e30-a2f5-037dde6c08f5" (UID: "04201ee2-1052-4e30-a2f5-037dde6c08f5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:02.492215 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.492169 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgtn6\" (UniqueName: \"kubernetes.io/projected/0ac163f9-7481-4312-9c8b-ffdebfd383b6-kube-api-access-hgtn6\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8\" (UID: \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" Nov 02 18:07:02.492401 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.492362 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ac163f9-7481-4312-9c8b-ffdebfd383b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8\" (UID: \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" Nov 02 18:07:02.492890 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.492862 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ac163f9-7481-4312-9c8b-ffdebfd383b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8\" (UID: \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" Nov 02 18:07:02.493228 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.493198 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac163f9-7481-4312-9c8b-ffdebfd383b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8\" (UID: \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" Nov 02 18:07:02.493441 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.493415 5601 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-provider-credentials\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:02.493534 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.493447 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7drmp\" (UniqueName: \"kubernetes.io/projected/04201ee2-1052-4e30-a2f5-037dde6c08f5-kube-api-access-7drmp\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:02.594939 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.594847 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgtn6\" (UniqueName: \"kubernetes.io/projected/0ac163f9-7481-4312-9c8b-ffdebfd383b6-kube-api-access-hgtn6\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8\" (UID: \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" Nov 02 18:07:02.595119 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.594939 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ac163f9-7481-4312-9c8b-ffdebfd383b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8\" (UID: \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" Nov 02 18:07:02.595325 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.595284 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ac163f9-7481-4312-9c8b-ffdebfd383b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8\" (UID: \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" Nov 02 18:07:02.595437 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.595398 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac163f9-7481-4312-9c8b-ffdebfd383b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8\" (UID: \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" Nov 02 18:07:02.596114 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.596087 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ac163f9-7481-4312-9c8b-ffdebfd383b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8\" (UID: \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" Nov 02 18:07:02.596238 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.596218 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac163f9-7481-4312-9c8b-ffdebfd383b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8\" (UID: \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" Nov 02 18:07:02.597582 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.597558 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ac163f9-7481-4312-9c8b-ffdebfd383b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8\" (UID: \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" Nov 02 18:07:02.603832 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.603809 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgtn6\" (UniqueName: \"kubernetes.io/projected/0ac163f9-7481-4312-9c8b-ffdebfd383b6-kube-api-access-hgtn6\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8\" (UID: \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" Nov 02 18:07:02.673729 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:02.673695 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" Nov 02 18:07:03.032818 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:03.032786 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8"] Nov 02 18:07:03.033173 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:07:03.032957 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac163f9_7481_4312_9c8b_ffdebfd383b6.slice/crio-0f4a275b53769f56b1650de4f777f3b8744645ff70a3fadc07a5e54f78b9a34b WatchSource:0}: Error finding container 0f4a275b53769f56b1650de4f777f3b8744645ff70a3fadc07a5e54f78b9a34b: Status 404 returned error can't find the container with id 0f4a275b53769f56b1650de4f777f3b8744645ff70a3fadc07a5e54f78b9a34b Nov 02 18:07:03.244962 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:03.244924 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" event={"ID":"04201ee2-1052-4e30-a2f5-037dde6c08f5","Type":"ContainerDied","Data":"f41f47cfb9e6ed2793c9b0ffcf25174a942c0636b91b9f52649e872cc1f33ff9"} Nov 02 18:07:03.244962 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:03.244945 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm" Nov 02 18:07:03.245207 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:03.244997 5601 scope.go:122] "RemoveContainer" containerID="78c7230918e71329b27692f92fb5d6a2c5779eb5e47cd716aac4b9a9fa8cb94a" Nov 02 18:07:03.247366 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:03.247336 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" event={"ID":"0ac163f9-7481-4312-9c8b-ffdebfd383b6","Type":"ContainerStarted","Data":"d83ad43f7dbe4bba304320575f0b9427d58dffd3a7a12a97c79bbd3c77ffc07d"} Nov 02 18:07:03.247486 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:03.247398 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" Nov 02 18:07:03.247486 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:03.247423 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" event={"ID":"0ac163f9-7481-4312-9c8b-ffdebfd383b6","Type":"ContainerStarted","Data":"0f4a275b53769f56b1650de4f777f3b8744645ff70a3fadc07a5e54f78b9a34b"} Nov 02 18:07:03.270338 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:03.270281 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" podStartSLOduration=3.270267426 podStartE2EDuration="3.270267426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:03.267796403 +0000 UTC m=+1987.958850417" watchObservedRunningTime="2026-11-02 18:07:03.270267426 +0000 UTC m=+1987.961321442" Nov 02 18:07:03.296130 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:03.295990 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm"] Nov 02 18:07:03.303114 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:03.303074 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7f58mmsnm"] Nov 02 18:07:03.429278 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:03.429241 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04201ee2-1052-4e30-a2f5-037dde6c08f5" path="/var/lib/kubelet/pods/04201ee2-1052-4e30-a2f5-037dde6c08f5/volumes" Nov 02 18:07:04.132888 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:07:04.132852 5601 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-327608/agc-cred-6ce668d70ed3: secret "agc-cred-6ce668d70ed3" not found Nov 02 18:07:04.132888 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:07:04.132890 5601 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w: secret "agc-cred-6ce668d70ed3" not found Nov 02 18:07:04.133306 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:07:04.132958 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-provider-credentials podName:924f6e47-49a4-4084-a49a-e3717b2dad87 nodeName:}" failed. No retries permitted until 2026-11-02 18:07:20.132937966 +0000 UTC m=+2004.823991958 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" (UID: "924f6e47-49a4-4084-a49a-e3717b2dad87") : secret "agc-cred-6ce668d70ed3" not found Nov 02 18:07:04.710651 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:07:04.703735 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:07:04.878572 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:04.878528 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" podUID="1e43007a-4111-4b61-9497-268081a88b5b" containerName="payload-processing" containerID="cri-o://a0341775f81f6a5ba385df239f1a22dc1744a96926c982afd720768ac8bfbaed" gracePeriod=25 Nov 02 18:07:05.053582 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:05.052248 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" Nov 02 18:07:05.639816 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:05.597136 5601 generic.go:356] "Generic (PLEG): container finished" podID="1e43007a-4111-4b61-9497-268081a88b5b" containerID="a0341775f81f6a5ba385df239f1a22dc1744a96926c982afd720768ac8bfbaed" exitCode=0 Nov 02 18:07:05.650190 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:05.650125 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" event={"ID":"1e43007a-4111-4b61-9497-268081a88b5b","Type":"ContainerDied","Data":"a0341775f81f6a5ba385df239f1a22dc1744a96926c982afd720768ac8bfbaed"} Nov 02 18:07:05.687645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:05.684351 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w"] Nov 02 18:07:05.743079 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:05.743047 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" Nov 02 18:07:05.795604 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:05.795568 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e43007a-4111-4b61-9497-268081a88b5b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e43007a-4111-4b61-9497-268081a88b5b-routing-overlay-volume\") pod \"1e43007a-4111-4b61-9497-268081a88b5b\" (UID: \"1e43007a-4111-4b61-9497-268081a88b5b\") " Nov 02 18:07:05.795856 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:05.795838 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e43007a-4111-4b61-9497-268081a88b5b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e43007a-4111-4b61-9497-268081a88b5b-provider-credentials\") pod \"1e43007a-4111-4b61-9497-268081a88b5b\" (UID: \"1e43007a-4111-4b61-9497-268081a88b5b\") " Nov 02 18:07:05.795943 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:05.795883 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e43007a-4111-4b61-9497-268081a88b5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e43007a-4111-4b61-9497-268081a88b5b-plugins-config-volume\") pod \"1e43007a-4111-4b61-9497-268081a88b5b\" (UID: \"1e43007a-4111-4b61-9497-268081a88b5b\") " Nov 02 18:07:05.796005 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:05.795974 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e43007a-4111-4b61-9497-268081a88b5b-routing-overlay-volume" pod "1e43007a-4111-4b61-9497-268081a88b5b" (UID: "1e43007a-4111-4b61-9497-268081a88b5b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:05.796075 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:05.796002 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e43007a-4111-4b61-9497-268081a88b5b-kube-api-access-7bx5p\" (UniqueName: \"kubernetes.io/projected/1e43007a-4111-4b61-9497-268081a88b5b-kube-api-access-7bx5p\") pod \"1e43007a-4111-4b61-9497-268081a88b5b\" (UID: \"1e43007a-4111-4b61-9497-268081a88b5b\") " Nov 02 18:07:05.796329 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:05.796303 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e43007a-4111-4b61-9497-268081a88b5b-plugins-config-volume" pod "1e43007a-4111-4b61-9497-268081a88b5b" (UID: "1e43007a-4111-4b61-9497-268081a88b5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:05.796824 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:05.796800 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e43007a-4111-4b61-9497-268081a88b5b-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:05.796974 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:05.796828 5601 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e43007a-4111-4b61-9497-268081a88b5b-routing-overlay-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:05.798507 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:05.798480 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e43007a-4111-4b61-9497-268081a88b5b-provider-credentials" pod "1e43007a-4111-4b61-9497-268081a88b5b" (UID: "1e43007a-4111-4b61-9497-268081a88b5b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:05.798583 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:05.798545 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e43007a-4111-4b61-9497-268081a88b5b-kube-api-access-7bx5p" pod "1e43007a-4111-4b61-9497-268081a88b5b" (UID: "1e43007a-4111-4b61-9497-268081a88b5b"). InnerVolumeSpecName "kube-api-access-7bx5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:05.907300 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:05.907205 5601 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e43007a-4111-4b61-9497-268081a88b5b-provider-credentials\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:05.907300 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:05.907240 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bx5p\" (UniqueName: \"kubernetes.io/projected/1e43007a-4111-4b61-9497-268081a88b5b-kube-api-access-7bx5p\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:06.635005 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:06.634971 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" Nov 02 18:07:06.635215 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:06.635051 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb" event={"ID":"1e43007a-4111-4b61-9497-268081a88b5b","Type":"ContainerDied","Data":"4602ed6e2ac3badb1862a4337cd117533203c98a3bb64b7151ad352afbfcf0f9"} Nov 02 18:07:06.635215 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:06.635131 5601 scope.go:122] "RemoveContainer" containerID="a0341775f81f6a5ba385df239f1a22dc1744a96926c982afd720768ac8bfbaed" Nov 02 18:07:06.674743 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:06.674708 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb"] Nov 02 18:07:06.679113 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:06.679081 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6pshgb"] Nov 02 18:07:07.428861 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:07.428788 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e43007a-4111-4b61-9497-268081a88b5b" path="/var/lib/kubelet/pods/1e43007a-4111-4b61-9497-268081a88b5b/volumes" Nov 02 18:07:07.959063 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:07.959029 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b"] Nov 02 18:07:07.959654 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:07.959588 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" podUID="398ed64f-2082-405d-a5c3-3628b0fdd21f" containerName="istio-proxy" containerID="cri-o://8c6f42545013c742320559878b3b1e2dae9f592cd7e41915336ef23c5e656e7f" gracePeriod=30 Nov 02 18:07:09.508348 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:09.508180 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn"] Nov 02 18:07:10.167640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.164204 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb"] Nov 02 18:07:10.167640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.164561 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" podUID="62dd825e-e164-4e44-b5ec-4c003f89c113" containerName="istio-proxy" containerID="cri-o://2db54e9965a17344a6bb31b5f040421faa2a57c2fd457b420b585341937dd990" gracePeriod=30 Nov 02 18:07:10.376491 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.365739 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" podUID="0077aa8d-4c02-40d7-ac71-beb13830dcf9" containerName="istio-proxy" containerID="cri-o://bf6373fae3ef708428abe9a5b39e5b8beb0c501ebb2cc0dcb4fe3a3962790e12" gracePeriod=30 Nov 02 18:07:10.393950 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.393921 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78"] Nov 02 18:07:10.395277 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.395245 5601 cpu_manager.go:414] "containerMap: removing container" podUID="1e43007a-4111-4b61-9497-268081a88b5b" containerName="payload-processing" Nov 02 18:07:10.395469 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.395452 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e43007a-4111-4b61-9497-268081a88b5b" containerName="payload-processing" Nov 02 18:07:10.395798 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.395780 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e43007a-4111-4b61-9497-268081a88b5b" containerName="payload-processing" Nov 02 18:07:10.400172 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.400154 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78"] Nov 02 18:07:10.400340 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.400271 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" Nov 02 18:07:10.407255 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.407227 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"openshift-service-ca.crt\"" Nov 02 18:07:10.407474 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.407450 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"e2e-cfl-879b22-provider-cred\"" Nov 02 18:07:10.407584 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.407567 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"payload-processing-external-model-e2e--a4661749-dockercfg-vq976\"" Nov 02 18:07:10.408042 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.408023 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"kube-root-ca.crt\"" Nov 02 18:07:10.408490 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.408465 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a8e574\"" Nov 02 18:07:10.408703 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.408669 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"routing-overlay\"" Nov 02 18:07:10.508071 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.508017 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmjxw\" (UniqueName: \"kubernetes.io/projected/1b831430-b7e7-4e83-a59a-ebc46dcc8002-kube-api-access-gmjxw\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78\" (UID: \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" Nov 02 18:07:10.508252 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.508228 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b831430-b7e7-4e83-a59a-ebc46dcc8002-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78\" (UID: \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" Nov 02 18:07:10.508399 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.508359 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b831430-b7e7-4e83-a59a-ebc46dcc8002-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78\" (UID: \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" Nov 02 18:07:10.508878 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.508737 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b831430-b7e7-4e83-a59a-ebc46dcc8002-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78\" (UID: \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" Nov 02 18:07:10.610303 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.610244 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmjxw\" (UniqueName: \"kubernetes.io/projected/1b831430-b7e7-4e83-a59a-ebc46dcc8002-kube-api-access-gmjxw\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78\" (UID: \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" Nov 02 18:07:10.610646 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.610344 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b831430-b7e7-4e83-a59a-ebc46dcc8002-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78\" (UID: \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" Nov 02 18:07:10.610646 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.610444 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b831430-b7e7-4e83-a59a-ebc46dcc8002-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78\" (UID: \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" Nov 02 18:07:10.610646 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.610645 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b831430-b7e7-4e83-a59a-ebc46dcc8002-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78\" (UID: \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" Nov 02 18:07:10.611518 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.611489 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b831430-b7e7-4e83-a59a-ebc46dcc8002-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78\" (UID: \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" Nov 02 18:07:10.611695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.611597 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b831430-b7e7-4e83-a59a-ebc46dcc8002-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78\" (UID: \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" Nov 02 18:07:10.614034 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.614001 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b831430-b7e7-4e83-a59a-ebc46dcc8002-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78\" (UID: \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" Nov 02 18:07:10.619220 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.619196 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmjxw\" (UniqueName: \"kubernetes.io/projected/1b831430-b7e7-4e83-a59a-ebc46dcc8002-kube-api-access-gmjxw\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78\" (UID: \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" Nov 02 18:07:10.685022 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.684976 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" podUID="924f6e47-49a4-4084-a49a-e3717b2dad87" containerName="payload-processing" containerID="cri-o://d923d52b4d79814f26a17522ea42e228d3fed1bd579783ead76a3da2543ee82d" gracePeriod=25 Nov 02 18:07:10.717769 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:10.717706 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" Nov 02 18:07:11.010697 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.010295 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78"] Nov 02 18:07:11.017839 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.017804 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" Nov 02 18:07:11.019190 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.019163 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924f6e47-49a4-4084-a49a-e3717b2dad87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924f6e47-49a4-4084-a49a-e3717b2dad87-plugins-config-volume\") pod \"924f6e47-49a4-4084-a49a-e3717b2dad87\" (UID: \"924f6e47-49a4-4084-a49a-e3717b2dad87\") " Nov 02 18:07:11.019311 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.019242 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-provider-credentials\" (UniqueName: \"kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-provider-credentials\") pod \"924f6e47-49a4-4084-a49a-e3717b2dad87\" (UID: \"924f6e47-49a4-4084-a49a-e3717b2dad87\") " Nov 02 18:07:11.019374 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.019344 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-kube-api-access-x2jrx\" (UniqueName: \"kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-kube-api-access-x2jrx\") pod \"924f6e47-49a4-4084-a49a-e3717b2dad87\" (UID: \"924f6e47-49a4-4084-a49a-e3717b2dad87\") " Nov 02 18:07:11.028908 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.027947 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924f6e47-49a4-4084-a49a-e3717b2dad87-plugins-config-volume" pod "924f6e47-49a4-4084-a49a-e3717b2dad87" (UID: "924f6e47-49a4-4084-a49a-e3717b2dad87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:11.055687 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.054703 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924f6e47-49a4-4084-a49a-e3717b2dad87-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:11.065157 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.065092 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-provider-credentials" pod "924f6e47-49a4-4084-a49a-e3717b2dad87" (UID: "924f6e47-49a4-4084-a49a-e3717b2dad87"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:11.069941 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.069826 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-kube-api-access-x2jrx" pod "924f6e47-49a4-4084-a49a-e3717b2dad87" (UID: "924f6e47-49a4-4084-a49a-e3717b2dad87"). InnerVolumeSpecName "kube-api-access-x2jrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:11.155688 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.155659 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924f6e47-49a4-4084-a49a-e3717b2dad87-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/924f6e47-49a4-4084-a49a-e3717b2dad87-routing-overlay-volume\") pod \"924f6e47-49a4-4084-a49a-e3717b2dad87\" (UID: \"924f6e47-49a4-4084-a49a-e3717b2dad87\") " Nov 02 18:07:11.156005 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.155979 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924f6e47-49a4-4084-a49a-e3717b2dad87-routing-overlay-volume" pod "924f6e47-49a4-4084-a49a-e3717b2dad87" (UID: "924f6e47-49a4-4084-a49a-e3717b2dad87"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:11.156399 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.156383 5601 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-provider-credentials\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:11.156450 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.156402 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2jrx\" (UniqueName: \"kubernetes.io/projected/924f6e47-49a4-4084-a49a-e3717b2dad87-kube-api-access-x2jrx\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:11.156450 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.156413 5601 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/924f6e47-49a4-4084-a49a-e3717b2dad87-routing-overlay-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:11.373582 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.373486 5601 generic.go:356] "Generic (PLEG): container finished" podID="924f6e47-49a4-4084-a49a-e3717b2dad87" containerID="d923d52b4d79814f26a17522ea42e228d3fed1bd579783ead76a3da2543ee82d" exitCode=0 Nov 02 18:07:11.373777 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.373588 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" event={"ID":"924f6e47-49a4-4084-a49a-e3717b2dad87","Type":"ContainerDied","Data":"d923d52b4d79814f26a17522ea42e228d3fed1bd579783ead76a3da2543ee82d"} Nov 02 18:07:11.373777 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.373598 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" Nov 02 18:07:11.373777 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.373688 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w" event={"ID":"924f6e47-49a4-4084-a49a-e3717b2dad87","Type":"ContainerDied","Data":"05b8d8d8807c4471732fcde593c01ea28a6af7e0907f1bde4f64a4b72d791ec9"} Nov 02 18:07:11.373777 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.373721 5601 scope.go:122] "RemoveContainer" containerID="d923d52b4d79814f26a17522ea42e228d3fed1bd579783ead76a3da2543ee82d" Nov 02 18:07:11.375575 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.375547 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" event={"ID":"1b831430-b7e7-4e83-a59a-ebc46dcc8002","Type":"ContainerStarted","Data":"ebefb05e65d191ef5262830dfda3398357d9636b77184a449c9d80f0b274a1fe"} Nov 02 18:07:11.375717 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.375599 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" event={"ID":"1b831430-b7e7-4e83-a59a-ebc46dcc8002","Type":"ContainerStarted","Data":"981c791f44c2a8e0833f05f8aadbb6a3f0eb37e8c2c7623d7f701172c98fff6e"} Nov 02 18:07:11.375717 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.375673 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" Nov 02 18:07:11.384987 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.384963 5601 scope.go:122] "RemoveContainer" containerID="d923d52b4d79814f26a17522ea42e228d3fed1bd579783ead76a3da2543ee82d" Nov 02 18:07:11.385261 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:07:11.385242 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d923d52b4d79814f26a17522ea42e228d3fed1bd579783ead76a3da2543ee82d\": container with ID starting with d923d52b4d79814f26a17522ea42e228d3fed1bd579783ead76a3da2543ee82d not found: ID does not exist" containerID="d923d52b4d79814f26a17522ea42e228d3fed1bd579783ead76a3da2543ee82d" Nov 02 18:07:11.385324 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.385271 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d923d52b4d79814f26a17522ea42e228d3fed1bd579783ead76a3da2543ee82d"} err="failed to get container status \"d923d52b4d79814f26a17522ea42e228d3fed1bd579783ead76a3da2543ee82d\": rpc error: code = NotFound desc = could not find container \"d923d52b4d79814f26a17522ea42e228d3fed1bd579783ead76a3da2543ee82d\": container with ID starting with d923d52b4d79814f26a17522ea42e228d3fed1bd579783ead76a3da2543ee82d not found: ID does not exist" Nov 02 18:07:11.396117 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.396076 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" podStartSLOduration=1.3960640930000001 podStartE2EDuration="1.396064093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:11.393742081 +0000 UTC m=+1996.084796093" watchObservedRunningTime="2026-11-02 18:07:11.396064093 +0000 UTC m=+1996.087118104" Nov 02 18:07:11.416076 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.416048 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w"] Nov 02 18:07:11.429434 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:11.429411 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-5c557f84w"] Nov 02 18:07:13.342669 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.342644 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 18:07:13.379607 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.379570 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw"] Nov 02 18:07:13.380862 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.380833 5601 cpu_manager.go:414] "containerMap: removing container" podUID="924f6e47-49a4-4084-a49a-e3717b2dad87" containerName="payload-processing" Nov 02 18:07:13.380862 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.380861 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924f6e47-49a4-4084-a49a-e3717b2dad87" containerName="payload-processing" Nov 02 18:07:13.381108 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.380954 5601 cpu_manager.go:414] "containerMap: removing container" podUID="398ed64f-2082-405d-a5c3-3628b0fdd21f" containerName="istio-proxy" Nov 02 18:07:13.381108 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.380964 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="398ed64f-2082-405d-a5c3-3628b0fdd21f" containerName="istio-proxy" Nov 02 18:07:13.381198 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.381123 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="398ed64f-2082-405d-a5c3-3628b0fdd21f" containerName="istio-proxy" Nov 02 18:07:13.381198 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.381137 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="924f6e47-49a4-4084-a49a-e3717b2dad87" containerName="payload-processing" Nov 02 18:07:13.384296 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.384277 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.402457 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.402426 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-bee8f6-openshift-default-dockercfg-nxqnx\"" Nov 02 18:07:13.405832 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.405277 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-workload-socket\") pod \"398ed64f-2082-405d-a5c3-3628b0fdd21f\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " Nov 02 18:07:13.405971 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.405952 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" Nov 02 18:07:13.406102 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.405920 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" event={"ID":"398ed64f-2082-405d-a5c3-3628b0fdd21f","Type":"ContainerDied","Data":"8c6f42545013c742320559878b3b1e2dae9f592cd7e41915336ef23c5e656e7f"} Nov 02 18:07:13.406177 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.406139 5601 scope.go:122] "RemoveContainer" containerID="8c6f42545013c742320559878b3b1e2dae9f592cd7e41915336ef23c5e656e7f" Nov 02 18:07:13.406266 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.405892 5601 generic.go:356] "Generic (PLEG): container finished" podID="398ed64f-2082-405d-a5c3-3628b0fdd21f" containerID="8c6f42545013c742320559878b3b1e2dae9f592cd7e41915336ef23c5e656e7f" exitCode=0 Nov 02 18:07:13.406425 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.406398 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b" event={"ID":"398ed64f-2082-405d-a5c3-3628b0fdd21f","Type":"ContainerDied","Data":"da5639a2e8eb3145b3261cee143643102375f573405cc9f49f447b1b57f5c470"} Nov 02 18:07:13.406480 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.406461 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw"] Nov 02 18:07:13.406556 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.406532 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-workload-socket" pod "398ed64f-2082-405d-a5c3-3628b0fdd21f" (UID: "398ed64f-2082-405d-a5c3-3628b0fdd21f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:13.406718 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.406702 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398ed64f-2082-405d-a5c3-3628b0fdd21f-kube-api-access-4ck7w\" (UniqueName: \"kubernetes.io/projected/398ed64f-2082-405d-a5c3-3628b0fdd21f-kube-api-access-4ck7w\") pod \"398ed64f-2082-405d-a5c3-3628b0fdd21f\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " Nov 02 18:07:13.406784 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.406740 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-envoy\") pod \"398ed64f-2082-405d-a5c3-3628b0fdd21f\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " Nov 02 18:07:13.406838 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.406795 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-credential-socket\") pod \"398ed64f-2082-405d-a5c3-3628b0fdd21f\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " Nov 02 18:07:13.406921 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.406908 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/398ed64f-2082-405d-a5c3-3628b0fdd21f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398ed64f-2082-405d-a5c3-3628b0fdd21f-istiod-ca-cert\") pod \"398ed64f-2082-405d-a5c3-3628b0fdd21f\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " Nov 02 18:07:13.406977 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.406953 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-token\" (UniqueName: \"kubernetes.io/projected/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-token\") pod \"398ed64f-2082-405d-a5c3-3628b0fdd21f\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " Nov 02 18:07:13.407033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.406986 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-workload-certs\") pod \"398ed64f-2082-405d-a5c3-3628b0fdd21f\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " Nov 02 18:07:13.407087 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.407066 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-data\") pod \"398ed64f-2082-405d-a5c3-3628b0fdd21f\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " Nov 02 18:07:13.407171 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.407158 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-podinfo\") pod \"398ed64f-2082-405d-a5c3-3628b0fdd21f\" (UID: \"398ed64f-2082-405d-a5c3-3628b0fdd21f\") " Nov 02 18:07:13.407715 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.407688 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398ed64f-2082-405d-a5c3-3628b0fdd21f-istiod-ca-cert" pod "398ed64f-2082-405d-a5c3-3628b0fdd21f" (UID: "398ed64f-2082-405d-a5c3-3628b0fdd21f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:13.407945 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.407923 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:13.408008 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.407951 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398ed64f-2082-405d-a5c3-3628b0fdd21f-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:13.408161 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.408136 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-workload-certs" pod "398ed64f-2082-405d-a5c3-3628b0fdd21f" (UID: "398ed64f-2082-405d-a5c3-3628b0fdd21f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:13.409396 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.409371 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-data" pod "398ed64f-2082-405d-a5c3-3628b0fdd21f" (UID: "398ed64f-2082-405d-a5c3-3628b0fdd21f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:13.409594 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.409570 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-credential-socket" pod "398ed64f-2082-405d-a5c3-3628b0fdd21f" (UID: "398ed64f-2082-405d-a5c3-3628b0fdd21f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:13.411788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.411755 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-token" pod "398ed64f-2082-405d-a5c3-3628b0fdd21f" (UID: "398ed64f-2082-405d-a5c3-3628b0fdd21f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:13.412991 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.412965 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-envoy" pod "398ed64f-2082-405d-a5c3-3628b0fdd21f" (UID: "398ed64f-2082-405d-a5c3-3628b0fdd21f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:13.414290 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.414262 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-podinfo" pod "398ed64f-2082-405d-a5c3-3628b0fdd21f" (UID: "398ed64f-2082-405d-a5c3-3628b0fdd21f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:07:13.414705 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.414684 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398ed64f-2082-405d-a5c3-3628b0fdd21f-kube-api-access-4ck7w" pod "398ed64f-2082-405d-a5c3-3628b0fdd21f" (UID: "398ed64f-2082-405d-a5c3-3628b0fdd21f"). InnerVolumeSpecName "kube-api-access-4ck7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:13.435491 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.435456 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924f6e47-49a4-4084-a49a-e3717b2dad87" path="/var/lib/kubelet/pods/924f6e47-49a4-4084-a49a-e3717b2dad87/volumes" Nov 02 18:07:13.451019 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.450992 5601 scope.go:122] "RemoveContainer" containerID="8c6f42545013c742320559878b3b1e2dae9f592cd7e41915336ef23c5e656e7f" Nov 02 18:07:13.451356 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:07:13.451335 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6f42545013c742320559878b3b1e2dae9f592cd7e41915336ef23c5e656e7f\": container with ID starting with 8c6f42545013c742320559878b3b1e2dae9f592cd7e41915336ef23c5e656e7f not found: ID does not exist" containerID="8c6f42545013c742320559878b3b1e2dae9f592cd7e41915336ef23c5e656e7f" Nov 02 18:07:13.451415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.451366 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6f42545013c742320559878b3b1e2dae9f592cd7e41915336ef23c5e656e7f"} err="failed to get container status \"8c6f42545013c742320559878b3b1e2dae9f592cd7e41915336ef23c5e656e7f\": rpc error: code = NotFound desc = could not find container \"8c6f42545013c742320559878b3b1e2dae9f592cd7e41915336ef23c5e656e7f\": container with ID starting with 8c6f42545013c742320559878b3b1e2dae9f592cd7e41915336ef23c5e656e7f not found: ID does not exist" Nov 02 18:07:13.509854 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.509805 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfca870-149d-4258-b915-a50e119a5539-istiod-ca-cert\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.510048 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.510021 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-credential-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.510114 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.510074 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-workload-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.510184 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.510116 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxf6j\" (UniqueName: \"kubernetes.io/projected/1bfca870-149d-4258-b915-a50e119a5539-kube-api-access-dxf6j\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.510290 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.510276 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-istio-envoy\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.510376 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.510301 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfca870-149d-4258-b915-a50e119a5539-istio-podinfo\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.510376 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.510331 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-istio-data\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.510376 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.510370 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfca870-149d-4258-b915-a50e119a5539-istio-token\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.510493 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.510454 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-workload-certs\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.510655 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.510607 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ck7w\" (UniqueName: \"kubernetes.io/projected/398ed64f-2082-405d-a5c3-3628b0fdd21f-kube-api-access-4ck7w\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:13.510655 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.510637 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:13.510655 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.510650 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:13.510792 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.510663 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:13.510792 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.510677 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:13.510792 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.510690 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:13.510792 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.510703 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398ed64f-2082-405d-a5c3-3628b0fdd21f-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:13.611722 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.611637 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfca870-149d-4258-b915-a50e119a5539-istio-token\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.611722 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.611684 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-workload-certs\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.611957 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.611776 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfca870-149d-4258-b915-a50e119a5539-istiod-ca-cert\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.611957 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.611859 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-credential-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.611957 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.611885 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-workload-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.611957 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.611916 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxf6j\" (UniqueName: \"kubernetes.io/projected/1bfca870-149d-4258-b915-a50e119a5539-kube-api-access-dxf6j\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.612179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.612117 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-istio-envoy\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.612179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.612137 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-workload-certs\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.612179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.612148 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfca870-149d-4258-b915-a50e119a5539-istio-podinfo\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.612349 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.612244 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-istio-data\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.612588 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.612559 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-istio-data\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.612752 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.612715 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-workload-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.612883 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.612836 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfca870-149d-4258-b915-a50e119a5539-istiod-ca-cert\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.612883 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.612871 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-credential-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.614579 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.614551 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-istio-envoy\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.615139 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.615121 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfca870-149d-4258-b915-a50e119a5539-istio-podinfo\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.621112 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.621090 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfca870-149d-4258-b915-a50e119a5539-istio-token\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.621194 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.621160 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxf6j\" (UniqueName: \"kubernetes.io/projected/1bfca870-149d-4258-b915-a50e119a5539-kube-api-access-dxf6j\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.715436 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.715407 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:13.738462 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.738390 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b"] Nov 02 18:07:13.744771 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:13.744738 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-h8h4b"] Nov 02 18:07:14.095675 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:07:14.095638 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bfca870_149d_4258_b915_a50e119a5539.slice/crio-3929788826e02f2369d71eeb2f255eab983c8447dd5e9e0ae47cb2507f05ec15 WatchSource:0}: Error finding container 3929788826e02f2369d71eeb2f255eab983c8447dd5e9e0ae47cb2507f05ec15: Status 404 returned error can't find the container with id 3929788826e02f2369d71eeb2f255eab983c8447dd5e9e0ae47cb2507f05ec15 Nov 02 18:07:14.097758 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:14.097736 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw"] Nov 02 18:07:14.099289 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:14.099252 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:07:14.099367 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:14.099324 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:07:14.099367 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:14.099356 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:07:14.260398 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:14.260362 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" Nov 02 18:07:14.419855 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:14.419812 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" event={"ID":"1bfca870-149d-4258-b915-a50e119a5539","Type":"ContainerStarted","Data":"18f55657f124542e9f8b7f48f8d6cff3948bf87e8c68601899ae65e704df337e"} Nov 02 18:07:14.420281 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:14.419886 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" event={"ID":"1bfca870-149d-4258-b915-a50e119a5539","Type":"ContainerStarted","Data":"3929788826e02f2369d71eeb2f255eab983c8447dd5e9e0ae47cb2507f05ec15"} Nov 02 18:07:14.446608 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:14.446558 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" podStartSLOduration=199.446543879 podStartE2EDuration="3m19.446543879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:14.441169372 +0000 UTC m=+1999.132223397" watchObservedRunningTime="2026-11-02 18:07:14.446543879 +0000 UTC m=+1999.137597891" Nov 02 18:07:14.716222 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:14.716141 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:15.484530 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:15.472858 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="398ed64f-2082-405d-a5c3-3628b0fdd21f" path="/var/lib/kubelet/pods/398ed64f-2082-405d-a5c3-3628b0fdd21f/volumes" Nov 02 18:07:15.496676 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:15.495955 5601 generic.go:356] "Generic (PLEG): container finished" podID="62dd825e-e164-4e44-b5ec-4c003f89c113" containerID="2db54e9965a17344a6bb31b5f040421faa2a57c2fd457b420b585341937dd990" exitCode=0 Nov 02 18:07:15.496676 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:15.496011 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" event={"ID":"62dd825e-e164-4e44-b5ec-4c003f89c113","Type":"ContainerDied","Data":"2db54e9965a17344a6bb31b5f040421faa2a57c2fd457b420b585341937dd990"} Nov 02 18:07:15.498852 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:15.498823 5601 generic.go:356] "Generic (PLEG): container finished" podID="0077aa8d-4c02-40d7-ac71-beb13830dcf9" containerID="bf6373fae3ef708428abe9a5b39e5b8beb0c501ebb2cc0dcb4fe3a3962790e12" exitCode=0 Nov 02 18:07:15.499035 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:15.499013 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" event={"ID":"0077aa8d-4c02-40d7-ac71-beb13830dcf9","Type":"ContainerDied","Data":"bf6373fae3ef708428abe9a5b39e5b8beb0c501ebb2cc0dcb4fe3a3962790e12"} Nov 02 18:07:15.714640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:15.706608 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 18:07:15.715046 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:15.714919 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" podUID="0077aa8d-4c02-40d7-ac71-beb13830dcf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 18:07:15.717832 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:15.717805 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 18:07:15.718101 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:15.718078 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" podUID="1bfca870-149d-4258-b915-a50e119a5539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 18:07:16.052935 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.052420 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 18:07:16.052935 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.052484 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" podUID="62dd825e-e164-4e44-b5ec-4c003f89c113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 18:07:16.738365 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.732472 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 18:07:16.738365 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.732525 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" podUID="1bfca870-149d-4258-b915-a50e119a5539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 18:07:16.802707 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.801571 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 18:07:16.934386 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.917470 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-envoy\") pod \"62dd825e-e164-4e44-b5ec-4c003f89c113\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " Nov 02 18:07:16.934386 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.917553 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62dd825e-e164-4e44-b5ec-4c003f89c113-kube-api-access-7m62h\" (UniqueName: \"kubernetes.io/projected/62dd825e-e164-4e44-b5ec-4c003f89c113-kube-api-access-7m62h\") pod \"62dd825e-e164-4e44-b5ec-4c003f89c113\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " Nov 02 18:07:16.934386 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.917682 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-workload-socket\") pod \"62dd825e-e164-4e44-b5ec-4c003f89c113\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " Nov 02 18:07:16.934386 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.917723 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-token\" (UniqueName: \"kubernetes.io/projected/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-token\") pod \"62dd825e-e164-4e44-b5ec-4c003f89c113\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " Nov 02 18:07:16.934386 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.917782 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62dd825e-e164-4e44-b5ec-4c003f89c113-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dd825e-e164-4e44-b5ec-4c003f89c113-istiod-ca-cert\") pod \"62dd825e-e164-4e44-b5ec-4c003f89c113\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " Nov 02 18:07:16.934386 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.917867 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-workload-certs\") pod \"62dd825e-e164-4e44-b5ec-4c003f89c113\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " Nov 02 18:07:16.934386 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.917948 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-data\") pod \"62dd825e-e164-4e44-b5ec-4c003f89c113\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " Nov 02 18:07:16.934386 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.918017 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-podinfo\") pod \"62dd825e-e164-4e44-b5ec-4c003f89c113\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " Nov 02 18:07:16.934386 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.918214 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-credential-socket\") pod \"62dd825e-e164-4e44-b5ec-4c003f89c113\" (UID: \"62dd825e-e164-4e44-b5ec-4c003f89c113\") " Nov 02 18:07:16.935702 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.935341 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-credential-socket" pod "62dd825e-e164-4e44-b5ec-4c003f89c113" (UID: "62dd825e-e164-4e44-b5ec-4c003f89c113"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:16.936000 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.935973 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62dd825e-e164-4e44-b5ec-4c003f89c113-istiod-ca-cert" pod "62dd825e-e164-4e44-b5ec-4c003f89c113" (UID: "62dd825e-e164-4e44-b5ec-4c003f89c113"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:16.936672 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.936379 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-data" pod "62dd825e-e164-4e44-b5ec-4c003f89c113" (UID: "62dd825e-e164-4e44-b5ec-4c003f89c113"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:16.943774 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.943705 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-workload-certs" pod "62dd825e-e164-4e44-b5ec-4c003f89c113" (UID: "62dd825e-e164-4e44-b5ec-4c003f89c113"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:16.965175 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.951869 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-workload-socket" pod "62dd825e-e164-4e44-b5ec-4c003f89c113" (UID: "62dd825e-e164-4e44-b5ec-4c003f89c113"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:17.000656 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.974954 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-envoy" pod "62dd825e-e164-4e44-b5ec-4c003f89c113" (UID: "62dd825e-e164-4e44-b5ec-4c003f89c113"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:17.000656 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:16.988797 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62dd825e-e164-4e44-b5ec-4c003f89c113-kube-api-access-7m62h" pod "62dd825e-e164-4e44-b5ec-4c003f89c113" (UID: "62dd825e-e164-4e44-b5ec-4c003f89c113"). InnerVolumeSpecName "kube-api-access-7m62h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:17.034680 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.029311 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:17.034680 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.029356 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m62h\" (UniqueName: \"kubernetes.io/projected/62dd825e-e164-4e44-b5ec-4c003f89c113-kube-api-access-7m62h\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:17.034680 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.029372 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:17.034680 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.029388 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dd825e-e164-4e44-b5ec-4c003f89c113-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:17.034680 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.029402 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:17.034680 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.029417 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:17.034680 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.029431 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dd825e-e164-4e44-b5ec-4c003f89c113-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:17.062712 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.060843 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 18:07:17.115460 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.115423 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh"] Nov 02 18:07:17.137545 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.137510 5601 cpu_manager.go:414] "containerMap: removing container" podUID="0077aa8d-4c02-40d7-ac71-beb13830dcf9" containerName="istio-proxy" Nov 02 18:07:17.148386 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.148360 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0077aa8d-4c02-40d7-ac71-beb13830dcf9" containerName="istio-proxy" Nov 02 18:07:17.148593 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.148576 5601 cpu_manager.go:414] "containerMap: removing container" podUID="62dd825e-e164-4e44-b5ec-4c003f89c113" containerName="istio-proxy" Nov 02 18:07:17.148944 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.148926 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62dd825e-e164-4e44-b5ec-4c003f89c113" containerName="istio-proxy" Nov 02 18:07:17.149331 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.149311 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="62dd825e-e164-4e44-b5ec-4c003f89c113" containerName="istio-proxy" Nov 02 18:07:17.149466 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.149454 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="0077aa8d-4c02-40d7-ac71-beb13830dcf9" containerName="istio-proxy" Nov 02 18:07:17.264960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.243678 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-credential-socket\") pod \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " Nov 02 18:07:17.264960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.243872 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0077aa8d-4c02-40d7-ac71-beb13830dcf9-kube-api-access-ngblj\" (UniqueName: \"kubernetes.io/projected/0077aa8d-4c02-40d7-ac71-beb13830dcf9-kube-api-access-ngblj\") pod \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " Nov 02 18:07:17.264960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.243944 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-podinfo\") pod \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " Nov 02 18:07:17.264960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.244064 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-workload-certs\") pod \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " Nov 02 18:07:17.264960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.244161 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-envoy\") pod \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " Nov 02 18:07:17.264960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.244242 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-workload-socket\") pod \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " Nov 02 18:07:17.264960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.244275 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-token\" (UniqueName: \"kubernetes.io/projected/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-token\") pod \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " Nov 02 18:07:17.264960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.244390 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istiod-ca-cert\") pod \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " Nov 02 18:07:17.264960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.244470 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-data\") pod \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\" (UID: \"0077aa8d-4c02-40d7-ac71-beb13830dcf9\") " Nov 02 18:07:17.264960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.245644 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-data" pod "0077aa8d-4c02-40d7-ac71-beb13830dcf9" (UID: "0077aa8d-4c02-40d7-ac71-beb13830dcf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:17.264960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.245865 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-credential-socket" pod "0077aa8d-4c02-40d7-ac71-beb13830dcf9" (UID: "0077aa8d-4c02-40d7-ac71-beb13830dcf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:17.370665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.294717 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-podinfo" pod "0077aa8d-4c02-40d7-ac71-beb13830dcf9" (UID: "0077aa8d-4c02-40d7-ac71-beb13830dcf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:07:17.370665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.296766 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-envoy" pod "0077aa8d-4c02-40d7-ac71-beb13830dcf9" (UID: "0077aa8d-4c02-40d7-ac71-beb13830dcf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:17.370665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.296995 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0077aa8d-4c02-40d7-ac71-beb13830dcf9-kube-api-access-ngblj" pod "0077aa8d-4c02-40d7-ac71-beb13830dcf9" (UID: "0077aa8d-4c02-40d7-ac71-beb13830dcf9"). InnerVolumeSpecName "kube-api-access-ngblj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:17.370665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.309734 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-workload-socket" pod "0077aa8d-4c02-40d7-ac71-beb13830dcf9" (UID: "0077aa8d-4c02-40d7-ac71-beb13830dcf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:17.370665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.313739 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-workload-certs" pod "0077aa8d-4c02-40d7-ac71-beb13830dcf9" (UID: "0077aa8d-4c02-40d7-ac71-beb13830dcf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:17.370665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.324755 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-token" pod "0077aa8d-4c02-40d7-ac71-beb13830dcf9" (UID: "0077aa8d-4c02-40d7-ac71-beb13830dcf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:17.370665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.340131 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6"] Nov 02 18:07:17.370665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.349659 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.370665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.352413 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:17.370665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.352442 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:17.370665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.352459 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:17.370665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.352476 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:17.370665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.352491 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:17.370665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.352505 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0077aa8d-4c02-40d7-ac71-beb13830dcf9-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:17.370665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.352520 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngblj\" (UniqueName: \"kubernetes.io/projected/0077aa8d-4c02-40d7-ac71-beb13830dcf9-kube-api-access-ngblj\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:17.370665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.352534 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:17.396801 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.395437 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh"] Nov 02 18:07:17.396801 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.395541 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" Nov 02 18:07:17.403817 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.401497 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6"] Nov 02 18:07:17.403817 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.402072 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5ec436\"/\"payload-processing-external-model-plugins-e2e-extmod-a-5ec436\"" Nov 02 18:07:17.403817 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.402877 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5ec436\"/\"kube-root-ca.crt\"" Nov 02 18:07:17.403817 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.402957 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istiod-ca-cert" pod "0077aa8d-4c02-40d7-ac71-beb13830dcf9" (UID: "0077aa8d-4c02-40d7-ac71-beb13830dcf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:17.403817 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.402974 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5ec436\"/\"openshift-service-ca.crt\"" Nov 02 18:07:17.403817 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.403059 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5ec436\"/\"routing-overlay\"" Nov 02 18:07:17.403817 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.403149 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-2db162-openshift-default-dockercfg-ms2fb\"" Nov 02 18:07:17.405442 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.405406 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-podinfo" pod "62dd825e-e164-4e44-b5ec-4c003f89c113" (UID: "62dd825e-e164-4e44-b5ec-4c003f89c113"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:07:17.419180 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.418232 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj"] Nov 02 18:07:17.419180 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.418442 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" podUID="686404fe-e7bb-4a29-9f7f-e07021328d8c" containerName="istio-proxy" containerID="cri-o://95370994daf483d00e49107340ad221f685658ac04fd8ea8b66ecece82bf9805" gracePeriod=30 Nov 02 18:07:17.476080 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.464176 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-envoy\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.476080 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.464263 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-podinfo\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.476080 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.464303 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-workload-certs\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.476080 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.464387 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-credential-socket\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.476080 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.464495 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-token\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.476080 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.464579 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-workload-socket\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.476080 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.464605 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns9cx\" (UniqueName: \"kubernetes.io/projected/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-kube-api-access-ns9cx\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.476080 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.464656 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istiod-ca-cert\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.476080 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.464761 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-data\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.476080 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.464896 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:17.476080 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.464914 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0077aa8d-4c02-40d7-ac71-beb13830dcf9-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:17.484874 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.484744 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5ec436\"/\"agc-cred-109ef9a24711\"" Nov 02 18:07:17.486136 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.486070 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5ec436\"/\"agc-cred-c13e2e0d36b2\"" Nov 02 18:07:17.486659 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.486442 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5ec436\"/\"payload-processing-external-model-e2e--96dec6bd-dockercfg-qbg29\"" Nov 02 18:07:17.504744 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.502745 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-token" pod "62dd825e-e164-4e44-b5ec-4c003f89c113" (UID: "62dd825e-e164-4e44-b5ec-4c003f89c113"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:17.515230 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.513946 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj"] Nov 02 18:07:17.520504 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.519832 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj"] Nov 02 18:07:17.520504 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.519950 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.575950 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.574317 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-data\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.575950 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.574415 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0db86edb-4352-4d70-a99f-4070d736fb52-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6\" (UID: \"0db86edb-4352-4d70-a99f-4070d736fb52\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" Nov 02 18:07:17.575950 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.574521 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0db86edb-4352-4d70-a99f-4070d736fb52-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6\" (UID: \"0db86edb-4352-4d70-a99f-4070d736fb52\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" Nov 02 18:07:17.575950 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.574632 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db86edb-4352-4d70-a99f-4070d736fb52-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6\" (UID: \"0db86edb-4352-4d70-a99f-4070d736fb52\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" Nov 02 18:07:17.575950 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.574718 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-envoy\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.575950 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.574794 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-podinfo\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.575950 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.574834 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-workload-certs\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.575950 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.574880 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-credential-socket\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.575950 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.574999 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-token\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.575950 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.575101 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-workload-socket\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.575950 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.575128 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns9cx\" (UniqueName: \"kubernetes.io/projected/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-kube-api-access-ns9cx\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.575950 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.575169 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istiod-ca-cert\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.575950 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.575201 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz5vp\" (UniqueName: \"kubernetes.io/projected/0db86edb-4352-4d70-a99f-4070d736fb52-kube-api-access-zz5vp\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6\" (UID: \"0db86edb-4352-4d70-a99f-4070d736fb52\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" Nov 02 18:07:17.575950 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.575318 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62dd825e-e164-4e44-b5ec-4c003f89c113-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:17.583257 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.580984 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istiod-ca-cert\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.583257 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.581214 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-workload-certs\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.583257 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.581386 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-data\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.583257 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.582172 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-workload-socket\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.583257 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.582371 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-credential-socket\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.587045 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.584906 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-envoy\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.591493 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.591460 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-token\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.599596 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.599571 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns9cx\" (UniqueName: \"kubernetes.io/projected/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-kube-api-access-ns9cx\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.601715 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.601687 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-podinfo\") pod \"e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.677096 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.677052 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db86edb-4352-4d70-a99f-4070d736fb52-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6\" (UID: \"0db86edb-4352-4d70-a99f-4070d736fb52\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" Nov 02 18:07:17.677280 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.677154 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-workload-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.677347 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.677313 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fa2df6-995c-4f63-b100-8dcec94085d7-istiod-ca-cert\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.677445 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.677427 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-data\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.677507 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.677458 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qr7m\" (UniqueName: \"kubernetes.io/projected/d8fa2df6-995c-4f63-b100-8dcec94085d7-kube-api-access-9qr7m\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.677562 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.677504 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-podinfo\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.677562 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.677545 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-credential-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.677730 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.677651 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz5vp\" (UniqueName: \"kubernetes.io/projected/0db86edb-4352-4d70-a99f-4070d736fb52-kube-api-access-zz5vp\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6\" (UID: \"0db86edb-4352-4d70-a99f-4070d736fb52\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" Nov 02 18:07:17.677730 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.677721 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-envoy\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.677837 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.677805 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-workload-certs\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.677910 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.677890 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0db86edb-4352-4d70-a99f-4070d736fb52-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6\" (UID: \"0db86edb-4352-4d70-a99f-4070d736fb52\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" Nov 02 18:07:17.678102 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.678080 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-token\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.678236 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.678218 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0db86edb-4352-4d70-a99f-4070d736fb52-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6\" (UID: \"0db86edb-4352-4d70-a99f-4070d736fb52\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" Nov 02 18:07:17.679695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.679669 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db86edb-4352-4d70-a99f-4070d736fb52-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6\" (UID: \"0db86edb-4352-4d70-a99f-4070d736fb52\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" Nov 02 18:07:17.680160 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.680133 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0db86edb-4352-4d70-a99f-4070d736fb52-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6\" (UID: \"0db86edb-4352-4d70-a99f-4070d736fb52\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" Nov 02 18:07:17.682778 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.682652 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0db86edb-4352-4d70-a99f-4070d736fb52-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6\" (UID: \"0db86edb-4352-4d70-a99f-4070d736fb52\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" Nov 02 18:07:17.690028 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.689946 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz5vp\" (UniqueName: \"kubernetes.io/projected/0db86edb-4352-4d70-a99f-4070d736fb52-kube-api-access-zz5vp\") pod \"payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6\" (UID: \"0db86edb-4352-4d70-a99f-4070d736fb52\") " pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" Nov 02 18:07:17.694417 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.694395 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" Nov 02 18:07:17.694647 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.694490 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb" event={"ID":"62dd825e-e164-4e44-b5ec-4c003f89c113","Type":"ContainerDied","Data":"c2ef3958e272c7823401fbb9837210558a036cda029cc78f028586519f13e3b9"} Nov 02 18:07:17.694751 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.694722 5601 scope.go:122] "RemoveContainer" containerID="2db54e9965a17344a6bb31b5f040421faa2a57c2fd457b420b585341937dd990" Nov 02 18:07:17.696819 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.696792 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" Nov 02 18:07:17.696905 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.696816 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn" event={"ID":"0077aa8d-4c02-40d7-ac71-beb13830dcf9","Type":"ContainerDied","Data":"a16455c8b47a0abb4c8c39b89f231e8c3591a57b80dcf7e059d5d04ef7dde28f"} Nov 02 18:07:17.700010 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.699987 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:17.717125 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.717097 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 18:07:17.717256 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.717167 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" podUID="1bfca870-149d-4258-b915-a50e119a5539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 18:07:17.718987 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.718964 5601 scope.go:122] "RemoveContainer" containerID="bf6373fae3ef708428abe9a5b39e5b8beb0c501ebb2cc0dcb4fe3a3962790e12" Nov 02 18:07:17.780085 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.780044 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-data\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.780540 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.780093 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qr7m\" (UniqueName: \"kubernetes.io/projected/d8fa2df6-995c-4f63-b100-8dcec94085d7-kube-api-access-9qr7m\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.780540 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.780127 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-podinfo\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.780540 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.780174 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-credential-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.780540 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.780281 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-envoy\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.780540 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.780349 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-workload-certs\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.780540 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.780414 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-token\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.780932 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.780594 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-workload-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.780932 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.780752 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fa2df6-995c-4f63-b100-8dcec94085d7-istiod-ca-cert\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.781040 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.780979 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-workload-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.781098 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.781067 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-data\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.781203 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.781090 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-workload-certs\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.781258 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.781222 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-credential-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.781941 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.781915 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fa2df6-995c-4f63-b100-8dcec94085d7-istiod-ca-cert\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.784263 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.784129 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-podinfo\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.784611 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.784593 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-envoy\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.796340 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.796310 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qr7m\" (UniqueName: \"kubernetes.io/projected/d8fa2df6-995c-4f63-b100-8dcec94085d7-kube-api-access-9qr7m\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.797808 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.797784 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb"] Nov 02 18:07:17.803349 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.803324 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-57wxb"] Nov 02 18:07:17.805543 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.805522 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" Nov 02 18:07:17.815307 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.812335 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-token\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:17.815307 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.815026 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn"] Nov 02 18:07:17.824299 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.824269 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-sd6xn"] Nov 02 18:07:17.886251 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:17.886211 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:18.031278 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.031249 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh"] Nov 02 18:07:18.058655 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.057259 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:07:18.058655 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.057364 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:07:18.058655 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.057411 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:07:18.144287 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:07:18.136640 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db86edb_4352_4d70_a99f_4070d736fb52.slice/crio-5cf3da93a97ce0af3f8f8ba35e1e4da3ece123ad6ff374d022b555b80f35a24c WatchSource:0}: Error finding container 5cf3da93a97ce0af3f8f8ba35e1e4da3ece123ad6ff374d022b555b80f35a24c: Status 404 returned error can't find the container with id 5cf3da93a97ce0af3f8f8ba35e1e4da3ece123ad6ff374d022b555b80f35a24c Nov 02 18:07:18.144287 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.139193 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6"] Nov 02 18:07:18.307658 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.303696 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj"] Nov 02 18:07:18.375839 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.375168 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:07:18.375839 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.375270 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:07:18.375839 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.375327 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:07:18.711628 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.711581 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" event={"ID":"0db86edb-4352-4d70-a99f-4070d736fb52","Type":"ContainerStarted","Data":"5e4b383778e99f7f9db3aab736a7c358534139d0c0a8f18a2797a6dabd21fec1"} Nov 02 18:07:18.711883 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.711857 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" event={"ID":"0db86edb-4352-4d70-a99f-4070d736fb52","Type":"ContainerStarted","Data":"5cf3da93a97ce0af3f8f8ba35e1e4da3ece123ad6ff374d022b555b80f35a24c"} Nov 02 18:07:18.712010 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.711996 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" Nov 02 18:07:18.713698 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.713671 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" event={"ID":"ad96fe9c-98b9-41bb-b532-1b3ca4da572c","Type":"ContainerStarted","Data":"9682a965c39fdc4f1051f46b8adf3adfe9b24ebca419ea815e91564751689a46"} Nov 02 18:07:18.713779 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.713717 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" event={"ID":"ad96fe9c-98b9-41bb-b532-1b3ca4da572c","Type":"ContainerStarted","Data":"9b3b6bd9f2c2c1aef557892795c21ac3670130fe1342b1add9f2610c3a9b4703"} Nov 02 18:07:18.716404 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.716381 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 18:07:18.716541 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.716512 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" event={"ID":"d8fa2df6-995c-4f63-b100-8dcec94085d7","Type":"ContainerStarted","Data":"f7c8a6edae06d9381e228bcadcc83b412f957343d048d7feabfd127af48321dd"} Nov 02 18:07:18.716680 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.716567 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" event={"ID":"d8fa2df6-995c-4f63-b100-8dcec94085d7","Type":"ContainerStarted","Data":"0636116adcc2488cba2ecd5a0256e13f632b82a326b435dd1e6cb68358733056"} Nov 02 18:07:18.716789 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.716763 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" podUID="1bfca870-149d-4258-b915-a50e119a5539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 18:07:18.786074 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.783935 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" podStartSLOduration=2.783915883 podStartE2EDuration="2.783915883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:18.773997532 +0000 UTC m=+2003.465051588" watchObservedRunningTime="2026-11-02 18:07:18.783915883 +0000 UTC m=+2003.474969896" Nov 02 18:07:18.811957 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.811897 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" podStartSLOduration=179.811882724 podStartE2EDuration="2m59.811882724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:18.800077066 +0000 UTC m=+2003.491131325" watchObservedRunningTime="2026-11-02 18:07:18.811882724 +0000 UTC m=+2003.502936736" Nov 02 18:07:18.850643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.850566 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" podStartSLOduration=158.850545635 podStartE2EDuration="2m38.850545635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:18.831638019 +0000 UTC m=+2003.522692029" watchObservedRunningTime="2026-11-02 18:07:18.850545635 +0000 UTC m=+2003.541599648" Nov 02 18:07:18.886994 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:18.886963 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:19.450638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:19.445711 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0077aa8d-4c02-40d7-ac71-beb13830dcf9" path="/var/lib/kubelet/pods/0077aa8d-4c02-40d7-ac71-beb13830dcf9/volumes" Nov 02 18:07:19.450638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:19.446548 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62dd825e-e164-4e44-b5ec-4c003f89c113" path="/var/lib/kubelet/pods/62dd825e-e164-4e44-b5ec-4c003f89c113/volumes" Nov 02 18:07:19.700552 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:19.700479 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:19.708207 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:19.708127 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:19.716174 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:19.716113 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 18:07:19.716323 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:19.716200 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" podUID="1bfca870-149d-4258-b915-a50e119a5539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 18:07:19.838640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:19.790221 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:19.896438 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:19.876682 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:07:19.896438 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:19.894125 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 02 18:07:19.896438 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:19.894203 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" podUID="d8fa2df6-995c-4f63-b100-8dcec94085d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 02 18:07:20.716889 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:20.716852 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 18:07:20.717075 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:20.716920 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" podUID="1bfca870-149d-4258-b915-a50e119a5539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 18:07:20.888643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:20.888478 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 02 18:07:20.888643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:20.888563 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" podUID="d8fa2df6-995c-4f63-b100-8dcec94085d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 02 18:07:21.803433 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:21.803400 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:21.804164 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:21.804141 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:21.805184 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:21.805162 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:07:21.889355 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:21.888760 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 02 18:07:21.889355 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:21.888817 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" podUID="d8fa2df6-995c-4f63-b100-8dcec94085d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 02 18:07:22.391878 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:22.391848 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" Nov 02 18:07:22.838806 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:22.837506 5601 generic.go:356] "Generic (PLEG): container finished" podID="686404fe-e7bb-4a29-9f7f-e07021328d8c" containerID="95370994daf483d00e49107340ad221f685658ac04fd8ea8b66ecece82bf9805" exitCode=0 Nov 02 18:07:22.838806 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:22.838734 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" event={"ID":"686404fe-e7bb-4a29-9f7f-e07021328d8c","Type":"ContainerDied","Data":"95370994daf483d00e49107340ad221f685658ac04fd8ea8b66ecece82bf9805"} Nov 02 18:07:22.913691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:22.906012 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 02 18:07:22.913691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:22.906074 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" podUID="d8fa2df6-995c-4f63-b100-8dcec94085d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 02 18:07:23.278702 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.274274 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 18:07:23.466669 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.466522 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-token\" (UniqueName: \"kubernetes.io/projected/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-token\") pod \"686404fe-e7bb-4a29-9f7f-e07021328d8c\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " Nov 02 18:07:23.466937 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.466897 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/686404fe-e7bb-4a29-9f7f-e07021328d8c-kube-api-access-rcnjz\" (UniqueName: \"kubernetes.io/projected/686404fe-e7bb-4a29-9f7f-e07021328d8c-kube-api-access-rcnjz\") pod \"686404fe-e7bb-4a29-9f7f-e07021328d8c\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " Nov 02 18:07:23.467170 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.467149 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-envoy\") pod \"686404fe-e7bb-4a29-9f7f-e07021328d8c\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " Nov 02 18:07:23.467250 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.467236 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-data\") pod \"686404fe-e7bb-4a29-9f7f-e07021328d8c\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " Nov 02 18:07:23.467377 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.467358 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-workload-socket\") pod \"686404fe-e7bb-4a29-9f7f-e07021328d8c\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " Nov 02 18:07:23.467453 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.467434 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/686404fe-e7bb-4a29-9f7f-e07021328d8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686404fe-e7bb-4a29-9f7f-e07021328d8c-istiod-ca-cert\") pod \"686404fe-e7bb-4a29-9f7f-e07021328d8c\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " Nov 02 18:07:23.467575 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.467558 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-workload-certs\") pod \"686404fe-e7bb-4a29-9f7f-e07021328d8c\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " Nov 02 18:07:23.467700 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.467599 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-credential-socket\") pod \"686404fe-e7bb-4a29-9f7f-e07021328d8c\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " Nov 02 18:07:23.467700 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.467686 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-podinfo\") pod \"686404fe-e7bb-4a29-9f7f-e07021328d8c\" (UID: \"686404fe-e7bb-4a29-9f7f-e07021328d8c\") " Nov 02 18:07:23.467862 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.467833 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-data" pod "686404fe-e7bb-4a29-9f7f-e07021328d8c" (UID: "686404fe-e7bb-4a29-9f7f-e07021328d8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:23.468281 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.468249 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/686404fe-e7bb-4a29-9f7f-e07021328d8c-istiod-ca-cert" pod "686404fe-e7bb-4a29-9f7f-e07021328d8c" (UID: "686404fe-e7bb-4a29-9f7f-e07021328d8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:23.469442 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.469412 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-workload-socket" pod "686404fe-e7bb-4a29-9f7f-e07021328d8c" (UID: "686404fe-e7bb-4a29-9f7f-e07021328d8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:23.471202 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.471001 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:23.471202 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.471037 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:23.471202 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.471066 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686404fe-e7bb-4a29-9f7f-e07021328d8c-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:23.471435 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.471268 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-workload-certs" pod "686404fe-e7bb-4a29-9f7f-e07021328d8c" (UID: "686404fe-e7bb-4a29-9f7f-e07021328d8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:23.471503 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.471438 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-token" pod "686404fe-e7bb-4a29-9f7f-e07021328d8c" (UID: "686404fe-e7bb-4a29-9f7f-e07021328d8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:23.473670 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.471662 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-podinfo" pod "686404fe-e7bb-4a29-9f7f-e07021328d8c" (UID: "686404fe-e7bb-4a29-9f7f-e07021328d8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:07:23.473670 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.471682 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-credential-socket" pod "686404fe-e7bb-4a29-9f7f-e07021328d8c" (UID: "686404fe-e7bb-4a29-9f7f-e07021328d8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:23.476185 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.475204 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/686404fe-e7bb-4a29-9f7f-e07021328d8c-kube-api-access-rcnjz" pod "686404fe-e7bb-4a29-9f7f-e07021328d8c" (UID: "686404fe-e7bb-4a29-9f7f-e07021328d8c"). InnerVolumeSpecName "kube-api-access-rcnjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:23.481073 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.481043 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-envoy" pod "686404fe-e7bb-4a29-9f7f-e07021328d8c" (UID: "686404fe-e7bb-4a29-9f7f-e07021328d8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:23.572946 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.572906 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:23.572946 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.572945 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:23.573186 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.572961 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:23.573186 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.572974 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:23.573186 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.572986 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcnjz\" (UniqueName: \"kubernetes.io/projected/686404fe-e7bb-4a29-9f7f-e07021328d8c-kube-api-access-rcnjz\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:23.573186 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.573002 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686404fe-e7bb-4a29-9f7f-e07021328d8c-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:23.872063 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.871978 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" Nov 02 18:07:23.872222 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.871979 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj" event={"ID":"686404fe-e7bb-4a29-9f7f-e07021328d8c","Type":"ContainerDied","Data":"f81f3f9f7ee9bbc065334ae84f8bf573afd2e2c291b3a0b4f67e91aca42a185c"} Nov 02 18:07:23.872222 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.872116 5601 scope.go:122] "RemoveContainer" containerID="95370994daf483d00e49107340ad221f685658ac04fd8ea8b66ecece82bf9805" Nov 02 18:07:23.887791 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.887766 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 02 18:07:23.887896 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.887820 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" podUID="d8fa2df6-995c-4f63-b100-8dcec94085d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 02 18:07:23.900327 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.900286 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj"] Nov 02 18:07:23.904449 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:23.904424 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-r7jgj"] Nov 02 18:07:24.887544 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:24.887511 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 02 18:07:24.887998 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:24.887573 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" podUID="d8fa2df6-995c-4f63-b100-8dcec94085d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 02 18:07:25.430940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:25.430909 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="686404fe-e7bb-4a29-9f7f-e07021328d8c" path="/var/lib/kubelet/pods/686404fe-e7bb-4a29-9f7f-e07021328d8c/volumes" Nov 02 18:07:26.011478 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:26.009479 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:26.011478 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:26.011061 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:26.012318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:26.012252 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:07:29.793242 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:29.793212 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" Nov 02 18:07:30.593507 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:30.593466 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb"] Nov 02 18:07:34.485502 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:07:34.485465 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:07:35.594560 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:35.594497 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" podUID="e802fac6-939b-45d5-8bf7-1070a9aa256f" containerName="payload-processing" containerID="cri-o://c8ab79e9d239efcb04ce9957f0de38ed9f074beec65ccb6e219a57504e74af2b" gracePeriod=25 Nov 02 18:07:35.885139 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:35.885107 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" Nov 02 18:07:35.954198 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:35.954144 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e802fac6-939b-45d5-8bf7-1070a9aa256f-kube-api-access-mw6wc\" (UniqueName: \"kubernetes.io/projected/e802fac6-939b-45d5-8bf7-1070a9aa256f-kube-api-access-mw6wc\") pod \"e802fac6-939b-45d5-8bf7-1070a9aa256f\" (UID: \"e802fac6-939b-45d5-8bf7-1070a9aa256f\") " Nov 02 18:07:35.954391 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:35.954309 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e802fac6-939b-45d5-8bf7-1070a9aa256f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e802fac6-939b-45d5-8bf7-1070a9aa256f-plugins-config-volume\") pod \"e802fac6-939b-45d5-8bf7-1070a9aa256f\" (UID: \"e802fac6-939b-45d5-8bf7-1070a9aa256f\") " Nov 02 18:07:35.954391 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:35.954378 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e802fac6-939b-45d5-8bf7-1070a9aa256f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e802fac6-939b-45d5-8bf7-1070a9aa256f-routing-overlay-volume\") pod \"e802fac6-939b-45d5-8bf7-1070a9aa256f\" (UID: \"e802fac6-939b-45d5-8bf7-1070a9aa256f\") " Nov 02 18:07:35.954531 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:35.954462 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e802fac6-939b-45d5-8bf7-1070a9aa256f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e802fac6-939b-45d5-8bf7-1070a9aa256f-provider-credentials\") pod \"e802fac6-939b-45d5-8bf7-1070a9aa256f\" (UID: \"e802fac6-939b-45d5-8bf7-1070a9aa256f\") " Nov 02 18:07:35.955071 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:35.955043 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e802fac6-939b-45d5-8bf7-1070a9aa256f-plugins-config-volume" pod "e802fac6-939b-45d5-8bf7-1070a9aa256f" (UID: "e802fac6-939b-45d5-8bf7-1070a9aa256f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:35.955510 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:35.955488 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e802fac6-939b-45d5-8bf7-1070a9aa256f-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:35.956110 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:35.956087 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e802fac6-939b-45d5-8bf7-1070a9aa256f-routing-overlay-volume" pod "e802fac6-939b-45d5-8bf7-1070a9aa256f" (UID: "e802fac6-939b-45d5-8bf7-1070a9aa256f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:35.969389 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:35.969274 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e802fac6-939b-45d5-8bf7-1070a9aa256f-kube-api-access-mw6wc" pod "e802fac6-939b-45d5-8bf7-1070a9aa256f" (UID: "e802fac6-939b-45d5-8bf7-1070a9aa256f"). InnerVolumeSpecName "kube-api-access-mw6wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:35.969952 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:35.969925 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e802fac6-939b-45d5-8bf7-1070a9aa256f-provider-credentials" pod "e802fac6-939b-45d5-8bf7-1070a9aa256f" (UID: "e802fac6-939b-45d5-8bf7-1070a9aa256f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:36.057013 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:36.056973 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw6wc\" (UniqueName: \"kubernetes.io/projected/e802fac6-939b-45d5-8bf7-1070a9aa256f-kube-api-access-mw6wc\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:36.057013 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:36.057009 5601 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e802fac6-939b-45d5-8bf7-1070a9aa256f-routing-overlay-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:36.057013 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:36.057020 5601 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e802fac6-939b-45d5-8bf7-1070a9aa256f-provider-credentials\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:07:36.106076 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:36.105985 5601 generic.go:356] "Generic (PLEG): container finished" podID="e802fac6-939b-45d5-8bf7-1070a9aa256f" containerID="c8ab79e9d239efcb04ce9957f0de38ed9f074beec65ccb6e219a57504e74af2b" exitCode=0 Nov 02 18:07:36.106076 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:36.106039 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" Nov 02 18:07:36.106279 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:36.106075 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" event={"ID":"e802fac6-939b-45d5-8bf7-1070a9aa256f","Type":"ContainerDied","Data":"c8ab79e9d239efcb04ce9957f0de38ed9f074beec65ccb6e219a57504e74af2b"} Nov 02 18:07:36.106279 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:36.106157 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb" event={"ID":"e802fac6-939b-45d5-8bf7-1070a9aa256f","Type":"ContainerDied","Data":"106eca8ea58a93c1a1635056cd2d12f0a9ceea917568b4b4656c70ca00daf84a"} Nov 02 18:07:36.106279 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:36.106181 5601 scope.go:122] "RemoveContainer" containerID="c8ab79e9d239efcb04ce9957f0de38ed9f074beec65ccb6e219a57504e74af2b" Nov 02 18:07:36.119072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:36.119056 5601 scope.go:122] "RemoveContainer" containerID="c8ab79e9d239efcb04ce9957f0de38ed9f074beec65ccb6e219a57504e74af2b" Nov 02 18:07:36.119331 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:07:36.119311 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8ab79e9d239efcb04ce9957f0de38ed9f074beec65ccb6e219a57504e74af2b\": container with ID starting with c8ab79e9d239efcb04ce9957f0de38ed9f074beec65ccb6e219a57504e74af2b not found: ID does not exist" containerID="c8ab79e9d239efcb04ce9957f0de38ed9f074beec65ccb6e219a57504e74af2b" Nov 02 18:07:36.119372 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:36.119339 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ab79e9d239efcb04ce9957f0de38ed9f074beec65ccb6e219a57504e74af2b"} err="failed to get container status \"c8ab79e9d239efcb04ce9957f0de38ed9f074beec65ccb6e219a57504e74af2b\": rpc error: code = NotFound desc = could not find container \"c8ab79e9d239efcb04ce9957f0de38ed9f074beec65ccb6e219a57504e74af2b\": container with ID starting with c8ab79e9d239efcb04ce9957f0de38ed9f074beec65ccb6e219a57504e74af2b not found: ID does not exist" Nov 02 18:07:36.133735 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:36.133713 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb"] Nov 02 18:07:36.139536 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:36.139515 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-327608/payload-processing-external-model-e2e-extmod-b-327608-9f9cqbpkb"] Nov 02 18:07:37.429273 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:37.429227 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e802fac6-939b-45d5-8bf7-1070a9aa256f" path="/var/lib/kubelet/pods/e802fac6-939b-45d5-8bf7-1070a9aa256f/volumes" Nov 02 18:07:58.087054 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.079498 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf"] Nov 02 18:07:58.087054 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.080649 5601 cpu_manager.go:414] "containerMap: removing container" podUID="686404fe-e7bb-4a29-9f7f-e07021328d8c" containerName="istio-proxy" Nov 02 18:07:58.087054 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.080667 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="686404fe-e7bb-4a29-9f7f-e07021328d8c" containerName="istio-proxy" Nov 02 18:07:58.087054 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.080725 5601 cpu_manager.go:414] "containerMap: removing container" podUID="e802fac6-939b-45d5-8bf7-1070a9aa256f" containerName="payload-processing" Nov 02 18:07:58.087054 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.080733 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e802fac6-939b-45d5-8bf7-1070a9aa256f" containerName="payload-processing" Nov 02 18:07:58.087054 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.080890 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="686404fe-e7bb-4a29-9f7f-e07021328d8c" containerName="istio-proxy" Nov 02 18:07:58.087054 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.080912 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="e802fac6-939b-45d5-8bf7-1070a9aa256f" containerName="payload-processing" Nov 02 18:07:58.093139 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.092963 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.099410 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.099377 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf"] Nov 02 18:07:58.175905 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.175873 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-42d786-openshift-default-dockercfg-dkzz6\"" Nov 02 18:07:58.214950 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.214917 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-credential-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.215115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.214968 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-data\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.215115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.215096 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-workload-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.215232 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.215153 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-594ql\" (UniqueName: \"kubernetes.io/projected/8d603436-a2a3-41f2-8554-c654ac4a9a45-kube-api-access-594ql\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.215317 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.215293 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d603436-a2a3-41f2-8554-c654ac4a9a45-istiod-ca-cert\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.215388 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.215376 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-token\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.215435 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.215411 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-envoy\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.215497 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.215450 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-podinfo\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.215660 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.215594 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-workload-certs\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.316847 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.316813 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-envoy\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.317041 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.316871 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-podinfo\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.317041 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.316959 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-workload-certs\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.317041 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.317028 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-credential-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.317230 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.317075 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-data\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.317230 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.317157 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-workload-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.317230 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.317221 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-594ql\" (UniqueName: \"kubernetes.io/projected/8d603436-a2a3-41f2-8554-c654ac4a9a45-kube-api-access-594ql\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.317400 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.317298 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d603436-a2a3-41f2-8554-c654ac4a9a45-istiod-ca-cert\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.317400 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.317350 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-token\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.317515 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.317399 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-workload-certs\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.317515 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.317483 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-credential-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.317657 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.317590 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-workload-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.317792 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.317769 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-data\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.318141 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.318110 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d603436-a2a3-41f2-8554-c654ac4a9a45-istiod-ca-cert\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.319969 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.319939 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-podinfo\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.321061 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.321036 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-envoy\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.327526 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.327496 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-594ql\" (UniqueName: \"kubernetes.io/projected/8d603436-a2a3-41f2-8554-c654ac4a9a45-kube-api-access-594ql\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.327710 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.327687 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-token\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-rdssf\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.414895 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.414813 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:58.744793 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:07:58.739327 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d603436_a2a3_41f2_8554_c654ac4a9a45.slice/crio-0c28148396e659f3e7f4ef032063840b858faddfac72b1823635223756ba9d76 WatchSource:0}: Error finding container 0c28148396e659f3e7f4ef032063840b858faddfac72b1823635223756ba9d76: Status 404 returned error can't find the container with id 0c28148396e659f3e7f4ef032063840b858faddfac72b1823635223756ba9d76 Nov 02 18:07:58.747791 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.747050 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf"] Nov 02 18:07:58.766829 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.761914 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:07:58.766829 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.762056 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:07:58.766829 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:58.762100 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:07:59.336546 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:59.336446 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" event={"ID":"8d603436-a2a3-41f2-8554-c654ac4a9a45","Type":"ContainerStarted","Data":"b9165dc79af6077d94eafb7e850e87f649f1903f4719bacdeef18de7b10eef95"} Nov 02 18:07:59.336546 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:59.336513 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" event={"ID":"8d603436-a2a3-41f2-8554-c654ac4a9a45","Type":"ContainerStarted","Data":"0c28148396e659f3e7f4ef032063840b858faddfac72b1823635223756ba9d76"} Nov 02 18:07:59.398555 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:59.398498 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" podStartSLOduration=1.398483892 podStartE2EDuration="1.398483892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:59.39663822 +0000 UTC m=+2044.087692233" watchObservedRunningTime="2026-11-02 18:07:59.398483892 +0000 UTC m=+2044.089537903" Nov 02 18:07:59.415922 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:59.415887 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:07:59.417342 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:59.417313 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 18:07:59.417479 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:07:59.417371 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" podUID="8d603436-a2a3-41f2-8554-c654ac4a9a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 18:08:00.429642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:00.426734 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 18:08:00.429642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:00.426810 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" podUID="8d603436-a2a3-41f2-8554-c654ac4a9a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 18:08:01.459008 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:01.458956 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 18:08:01.480225 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:01.479699 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" podUID="8d603436-a2a3-41f2-8554-c654ac4a9a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 18:08:02.415494 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:02.415451 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 18:08:02.415722 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:02.415533 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" podUID="8d603436-a2a3-41f2-8554-c654ac4a9a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 18:08:03.421636 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:03.416756 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 18:08:03.421636 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:03.416835 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" podUID="8d603436-a2a3-41f2-8554-c654ac4a9a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 18:08:04.415932 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:04.415895 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 18:08:04.416126 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:04.415959 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" podUID="8d603436-a2a3-41f2-8554-c654ac4a9a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 18:08:04.514154 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:08:04.514117 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:08:05.416062 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:05.416028 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 18:08:05.416257 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:05.416092 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" podUID="8d603436-a2a3-41f2-8554-c654ac4a9a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 18:08:06.727455 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:06.727423 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:08:06.728060 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:06.727796 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:08:06.728628 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:06.728589 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:08:13.628287 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:13.628254 5601 generic.go:356] "Generic (PLEG): container finished" podID="ad96fe9c-98b9-41bb-b532-1b3ca4da572c" containerID="9682a965c39fdc4f1051f46b8adf3adfe9b24ebca419ea815e91564751689a46" exitCode=0 Nov 02 18:08:13.628749 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:13.628312 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" event={"ID":"ad96fe9c-98b9-41bb-b532-1b3ca4da572c","Type":"ContainerDied","Data":"9682a965c39fdc4f1051f46b8adf3adfe9b24ebca419ea815e91564751689a46"} Nov 02 18:08:13.628861 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:13.628845 5601 scope.go:122] "RemoveContainer" containerID="9682a965c39fdc4f1051f46b8adf3adfe9b24ebca419ea815e91564751689a46" Nov 02 18:08:13.631169 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:13.631134 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:08:13.631303 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:13.631225 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:08:13.631303 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:13.631278 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:08:14.642476 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:14.642441 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" event={"ID":"ad96fe9c-98b9-41bb-b532-1b3ca4da572c","Type":"ContainerStarted","Data":"0ae24df59b04dd5729bb916723d67cbc9466e2fc783fe2a3df269adbbb2e2205"} Nov 02 18:08:14.709488 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:14.709454 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:08:14.731111 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:14.730779 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 02 18:08:14.731401 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:14.731370 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" podUID="ad96fe9c-98b9-41bb-b532-1b3ca4da572c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 02 18:08:15.700580 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:15.700545 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 02 18:08:15.701011 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:15.700607 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" podUID="ad96fe9c-98b9-41bb-b532-1b3ca4da572c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 02 18:08:16.701357 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:16.701322 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 02 18:08:16.701783 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:16.701376 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" podUID="ad96fe9c-98b9-41bb-b532-1b3ca4da572c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 02 18:08:17.700489 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:17.700456 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 02 18:08:17.701124 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:17.700804 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" podUID="ad96fe9c-98b9-41bb-b532-1b3ca4da572c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 02 18:08:18.701150 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:18.701116 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 02 18:08:18.701722 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:18.701184 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" podUID="ad96fe9c-98b9-41bb-b532-1b3ca4da572c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 02 18:08:19.700462 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:19.700426 5601 patch_prober.go:29] interesting pod/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 02 18:08:19.700788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:19.700491 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" podUID="ad96fe9c-98b9-41bb-b532-1b3ca4da572c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 02 18:08:19.777437 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:19.777400 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:08:20.705965 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:20.705933 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:08:20.714260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:20.714232 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:08:24.366919 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:24.366879 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94"] Nov 02 18:08:24.367364 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:24.367089 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" podUID="171d0002-0cae-4bb8-a398-9ba0da9f393f" containerName="istio-proxy" containerID="cri-o://87fbcd32e41cedf2ac27888124ecfd468ba798670c7474ecd87de360104224a3" gracePeriod=30 Nov 02 18:08:29.680025 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.679990 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 18:08:29.785744 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.785706 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-workload-socket\") pod \"171d0002-0cae-4bb8-a398-9ba0da9f393f\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " Nov 02 18:08:29.785941 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.785795 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/171d0002-0cae-4bb8-a398-9ba0da9f393f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171d0002-0cae-4bb8-a398-9ba0da9f393f-istiod-ca-cert\") pod \"171d0002-0cae-4bb8-a398-9ba0da9f393f\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " Nov 02 18:08:29.785941 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.785891 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-credential-socket\") pod \"171d0002-0cae-4bb8-a398-9ba0da9f393f\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " Nov 02 18:08:29.786071 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.785931 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-workload-socket" pod "171d0002-0cae-4bb8-a398-9ba0da9f393f" (UID: "171d0002-0cae-4bb8-a398-9ba0da9f393f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:29.786071 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.786009 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/171d0002-0cae-4bb8-a398-9ba0da9f393f-kube-api-access-ckfht\" (UniqueName: \"kubernetes.io/projected/171d0002-0cae-4bb8-a398-9ba0da9f393f-kube-api-access-ckfht\") pod \"171d0002-0cae-4bb8-a398-9ba0da9f393f\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " Nov 02 18:08:29.786071 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.786054 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-podinfo\") pod \"171d0002-0cae-4bb8-a398-9ba0da9f393f\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " Nov 02 18:08:29.786238 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.786110 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-workload-certs\") pod \"171d0002-0cae-4bb8-a398-9ba0da9f393f\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " Nov 02 18:08:29.786238 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.786190 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/171d0002-0cae-4bb8-a398-9ba0da9f393f-istiod-ca-cert" pod "171d0002-0cae-4bb8-a398-9ba0da9f393f" (UID: "171d0002-0cae-4bb8-a398-9ba0da9f393f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:08:29.786238 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.786184 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-credential-socket" pod "171d0002-0cae-4bb8-a398-9ba0da9f393f" (UID: "171d0002-0cae-4bb8-a398-9ba0da9f393f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:29.786385 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.786235 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-envoy\") pod \"171d0002-0cae-4bb8-a398-9ba0da9f393f\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " Nov 02 18:08:29.786385 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.786267 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-token\" (UniqueName: \"kubernetes.io/projected/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-token\") pod \"171d0002-0cae-4bb8-a398-9ba0da9f393f\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " Nov 02 18:08:29.786488 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.786401 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-data\") pod \"171d0002-0cae-4bb8-a398-9ba0da9f393f\" (UID: \"171d0002-0cae-4bb8-a398-9ba0da9f393f\") " Nov 02 18:08:29.786547 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.786494 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-workload-certs" pod "171d0002-0cae-4bb8-a398-9ba0da9f393f" (UID: "171d0002-0cae-4bb8-a398-9ba0da9f393f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:29.787157 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.787128 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-data" pod "171d0002-0cae-4bb8-a398-9ba0da9f393f" (UID: "171d0002-0cae-4bb8-a398-9ba0da9f393f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:29.787511 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.787487 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:29.787688 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.787660 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:29.787804 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.787692 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:29.787804 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.787707 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171d0002-0cae-4bb8-a398-9ba0da9f393f-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:29.787804 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.787722 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:29.788895 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.788874 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-podinfo" pod "171d0002-0cae-4bb8-a398-9ba0da9f393f" (UID: "171d0002-0cae-4bb8-a398-9ba0da9f393f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:08:29.789087 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.789064 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-token" pod "171d0002-0cae-4bb8-a398-9ba0da9f393f" (UID: "171d0002-0cae-4bb8-a398-9ba0da9f393f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:29.793973 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.793950 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171d0002-0cae-4bb8-a398-9ba0da9f393f-kube-api-access-ckfht" pod "171d0002-0cae-4bb8-a398-9ba0da9f393f" (UID: "171d0002-0cae-4bb8-a398-9ba0da9f393f"). InnerVolumeSpecName "kube-api-access-ckfht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:29.795766 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.795743 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-envoy" pod "171d0002-0cae-4bb8-a398-9ba0da9f393f" (UID: "171d0002-0cae-4bb8-a398-9ba0da9f393f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:29.874249 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.874207 5601 generic.go:356] "Generic (PLEG): container finished" podID="171d0002-0cae-4bb8-a398-9ba0da9f393f" containerID="87fbcd32e41cedf2ac27888124ecfd468ba798670c7474ecd87de360104224a3" exitCode=0 Nov 02 18:08:29.874711 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.874348 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" event={"ID":"171d0002-0cae-4bb8-a398-9ba0da9f393f","Type":"ContainerDied","Data":"87fbcd32e41cedf2ac27888124ecfd468ba798670c7474ecd87de360104224a3"} Nov 02 18:08:29.874926 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.874769 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" event={"ID":"171d0002-0cae-4bb8-a398-9ba0da9f393f","Type":"ContainerDied","Data":"fb7101df07518f54d722aa371a6b867b2cf0fc0a306e0088eb8b5298405865b0"} Nov 02 18:08:29.874926 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.874460 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94" Nov 02 18:08:29.874926 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.874826 5601 scope.go:122] "RemoveContainer" containerID="87fbcd32e41cedf2ac27888124ecfd468ba798670c7474ecd87de360104224a3" Nov 02 18:08:29.889222 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.889154 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:29.889222 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.889189 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:29.889222 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.889206 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckfht\" (UniqueName: \"kubernetes.io/projected/171d0002-0cae-4bb8-a398-9ba0da9f393f-kube-api-access-ckfht\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:29.889222 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.889221 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171d0002-0cae-4bb8-a398-9ba0da9f393f-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:29.891941 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.891917 5601 scope.go:122] "RemoveContainer" containerID="87fbcd32e41cedf2ac27888124ecfd468ba798670c7474ecd87de360104224a3" Nov 02 18:08:29.892452 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:08:29.892425 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87fbcd32e41cedf2ac27888124ecfd468ba798670c7474ecd87de360104224a3\": container with ID starting with 87fbcd32e41cedf2ac27888124ecfd468ba798670c7474ecd87de360104224a3 not found: ID does not exist" containerID="87fbcd32e41cedf2ac27888124ecfd468ba798670c7474ecd87de360104224a3" Nov 02 18:08:29.892560 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.892459 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87fbcd32e41cedf2ac27888124ecfd468ba798670c7474ecd87de360104224a3"} err="failed to get container status \"87fbcd32e41cedf2ac27888124ecfd468ba798670c7474ecd87de360104224a3\": rpc error: code = NotFound desc = could not find container \"87fbcd32e41cedf2ac27888124ecfd468ba798670c7474ecd87de360104224a3\": container with ID starting with 87fbcd32e41cedf2ac27888124ecfd468ba798670c7474ecd87de360104224a3 not found: ID does not exist" Nov 02 18:08:29.907671 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.907634 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94"] Nov 02 18:08:29.913293 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:29.913256 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-r6t94"] Nov 02 18:08:31.429384 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:31.429350 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="171d0002-0cae-4bb8-a398-9ba0da9f393f" path="/var/lib/kubelet/pods/171d0002-0cae-4bb8-a398-9ba0da9f393f/volumes" Nov 02 18:08:32.106855 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:32.106824 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m"] Nov 02 18:08:32.107096 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:32.107043 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" podUID="25416f1c-51e9-48fe-a6c7-4382172adaf5" containerName="istio-proxy" containerID="cri-o://b2e66dae7643aad11ab2cd6720cc4b6f43e5df78874a1bee1ff1ffc8bfa7da36" gracePeriod=30 Nov 02 18:08:34.485608 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:08:34.485572 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:08:37.414179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.414150 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 18:08:37.456366 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.456335 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-token\" (UniqueName: \"kubernetes.io/projected/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-token\") pod \"25416f1c-51e9-48fe-a6c7-4382172adaf5\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " Nov 02 18:08:37.456567 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.456380 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25416f1c-51e9-48fe-a6c7-4382172adaf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25416f1c-51e9-48fe-a6c7-4382172adaf5-istiod-ca-cert\") pod \"25416f1c-51e9-48fe-a6c7-4382172adaf5\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " Nov 02 18:08:37.456567 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.456423 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25416f1c-51e9-48fe-a6c7-4382172adaf5-kube-api-access-29kxn\" (UniqueName: \"kubernetes.io/projected/25416f1c-51e9-48fe-a6c7-4382172adaf5-kube-api-access-29kxn\") pod \"25416f1c-51e9-48fe-a6c7-4382172adaf5\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " Nov 02 18:08:37.456567 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.456518 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-podinfo\") pod \"25416f1c-51e9-48fe-a6c7-4382172adaf5\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " Nov 02 18:08:37.456796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.456588 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-credential-socket\") pod \"25416f1c-51e9-48fe-a6c7-4382172adaf5\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " Nov 02 18:08:37.456796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.456637 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-envoy\") pod \"25416f1c-51e9-48fe-a6c7-4382172adaf5\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " Nov 02 18:08:37.456796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.456673 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-workload-certs\") pod \"25416f1c-51e9-48fe-a6c7-4382172adaf5\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " Nov 02 18:08:37.456796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.456734 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-data\") pod \"25416f1c-51e9-48fe-a6c7-4382172adaf5\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " Nov 02 18:08:37.457010 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.456800 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-workload-socket\") pod \"25416f1c-51e9-48fe-a6c7-4382172adaf5\" (UID: \"25416f1c-51e9-48fe-a6c7-4382172adaf5\") " Nov 02 18:08:37.457010 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.456929 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25416f1c-51e9-48fe-a6c7-4382172adaf5-istiod-ca-cert" pod "25416f1c-51e9-48fe-a6c7-4382172adaf5" (UID: "25416f1c-51e9-48fe-a6c7-4382172adaf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:08:37.457512 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.457484 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-data" pod "25416f1c-51e9-48fe-a6c7-4382172adaf5" (UID: "25416f1c-51e9-48fe-a6c7-4382172adaf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:37.457710 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.457688 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-workload-certs" pod "25416f1c-51e9-48fe-a6c7-4382172adaf5" (UID: "25416f1c-51e9-48fe-a6c7-4382172adaf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:37.457799 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.457719 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25416f1c-51e9-48fe-a6c7-4382172adaf5-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:37.457799 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.457742 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:37.458252 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.458227 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-workload-socket" pod "25416f1c-51e9-48fe-a6c7-4382172adaf5" (UID: "25416f1c-51e9-48fe-a6c7-4382172adaf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:37.458736 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.458709 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-credential-socket" pod "25416f1c-51e9-48fe-a6c7-4382172adaf5" (UID: "25416f1c-51e9-48fe-a6c7-4382172adaf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:37.459472 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.459446 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-token" pod "25416f1c-51e9-48fe-a6c7-4382172adaf5" (UID: "25416f1c-51e9-48fe-a6c7-4382172adaf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:37.460149 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.460129 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-envoy" pod "25416f1c-51e9-48fe-a6c7-4382172adaf5" (UID: "25416f1c-51e9-48fe-a6c7-4382172adaf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:37.461096 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.461075 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-podinfo" pod "25416f1c-51e9-48fe-a6c7-4382172adaf5" (UID: "25416f1c-51e9-48fe-a6c7-4382172adaf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:08:37.462374 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.462355 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25416f1c-51e9-48fe-a6c7-4382172adaf5-kube-api-access-29kxn" pod "25416f1c-51e9-48fe-a6c7-4382172adaf5" (UID: "25416f1c-51e9-48fe-a6c7-4382172adaf5"). InnerVolumeSpecName "kube-api-access-29kxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:37.559707 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.559669 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:37.559707 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.559701 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29kxn\" (UniqueName: \"kubernetes.io/projected/25416f1c-51e9-48fe-a6c7-4382172adaf5-kube-api-access-29kxn\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:37.559707 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.559712 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:37.559960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.559721 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:37.559960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.559730 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:37.559960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.559738 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:37.559960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:37.559747 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25416f1c-51e9-48fe-a6c7-4382172adaf5-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:38.014717 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:38.014683 5601 generic.go:356] "Generic (PLEG): container finished" podID="25416f1c-51e9-48fe-a6c7-4382172adaf5" containerID="b2e66dae7643aad11ab2cd6720cc4b6f43e5df78874a1bee1ff1ffc8bfa7da36" exitCode=0 Nov 02 18:08:38.014919 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:38.014740 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" Nov 02 18:08:38.014919 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:38.014756 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" event={"ID":"25416f1c-51e9-48fe-a6c7-4382172adaf5","Type":"ContainerDied","Data":"b2e66dae7643aad11ab2cd6720cc4b6f43e5df78874a1bee1ff1ffc8bfa7da36"} Nov 02 18:08:38.014919 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:38.014815 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m" event={"ID":"25416f1c-51e9-48fe-a6c7-4382172adaf5","Type":"ContainerDied","Data":"8692c43d95e523e96abeb7d55a163ca7e0acb9a7f88a13a16c1d083a29ee9293"} Nov 02 18:08:38.014919 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:38.014840 5601 scope.go:122] "RemoveContainer" containerID="b2e66dae7643aad11ab2cd6720cc4b6f43e5df78874a1bee1ff1ffc8bfa7da36" Nov 02 18:08:38.026844 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:38.026816 5601 scope.go:122] "RemoveContainer" containerID="b2e66dae7643aad11ab2cd6720cc4b6f43e5df78874a1bee1ff1ffc8bfa7da36" Nov 02 18:08:38.027183 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:08:38.027158 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e66dae7643aad11ab2cd6720cc4b6f43e5df78874a1bee1ff1ffc8bfa7da36\": container with ID starting with b2e66dae7643aad11ab2cd6720cc4b6f43e5df78874a1bee1ff1ffc8bfa7da36 not found: ID does not exist" containerID="b2e66dae7643aad11ab2cd6720cc4b6f43e5df78874a1bee1ff1ffc8bfa7da36" Nov 02 18:08:38.027243 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:38.027192 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e66dae7643aad11ab2cd6720cc4b6f43e5df78874a1bee1ff1ffc8bfa7da36"} err="failed to get container status \"b2e66dae7643aad11ab2cd6720cc4b6f43e5df78874a1bee1ff1ffc8bfa7da36\": rpc error: code = NotFound desc = could not find container \"b2e66dae7643aad11ab2cd6720cc4b6f43e5df78874a1bee1ff1ffc8bfa7da36\": container with ID starting with b2e66dae7643aad11ab2cd6720cc4b6f43e5df78874a1bee1ff1ffc8bfa7da36 not found: ID does not exist" Nov 02 18:08:38.047134 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:38.047109 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m"] Nov 02 18:08:38.058993 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:38.058966 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-9lp7m"] Nov 02 18:08:39.430979 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:39.430947 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25416f1c-51e9-48fe-a6c7-4382172adaf5" path="/var/lib/kubelet/pods/25416f1c-51e9-48fe-a6c7-4382172adaf5/volumes" Nov 02 18:08:43.120990 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.120956 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc"] Nov 02 18:08:43.121797 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.121778 5601 cpu_manager.go:414] "containerMap: removing container" podUID="25416f1c-51e9-48fe-a6c7-4382172adaf5" containerName="istio-proxy" Nov 02 18:08:43.121797 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.121797 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25416f1c-51e9-48fe-a6c7-4382172adaf5" containerName="istio-proxy" Nov 02 18:08:43.121893 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.121818 5601 cpu_manager.go:414] "containerMap: removing container" podUID="171d0002-0cae-4bb8-a398-9ba0da9f393f" containerName="istio-proxy" Nov 02 18:08:43.121893 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.121823 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="171d0002-0cae-4bb8-a398-9ba0da9f393f" containerName="istio-proxy" Nov 02 18:08:43.121965 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.121934 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="25416f1c-51e9-48fe-a6c7-4382172adaf5" containerName="istio-proxy" Nov 02 18:08:43.121965 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.121943 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="171d0002-0cae-4bb8-a398-9ba0da9f393f" containerName="istio-proxy" Nov 02 18:08:43.127213 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.127195 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.138842 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.138814 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc"] Nov 02 18:08:43.146934 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.146907 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-workload-certs\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.147095 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.146958 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-credential-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.147095 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.147059 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-workload-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.147219 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.147190 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-token\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.147312 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.147295 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd72fbc-016a-43de-95f0-4d939b864aac-istiod-ca-cert\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.147379 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.147352 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpd4c\" (UniqueName: \"kubernetes.io/projected/8fd72fbc-016a-43de-95f0-4d939b864aac-kube-api-access-cpd4c\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.147487 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.147465 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-data\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.147544 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.147511 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-podinfo\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.147661 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.147644 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-envoy\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.248900 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.248865 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-data\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.249082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.248907 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-podinfo\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.249082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.248956 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-envoy\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.249082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.248997 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-workload-certs\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.249082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.249029 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-credential-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.249082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.249072 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-workload-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.249319 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.249158 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-token\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.249319 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.249269 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd72fbc-016a-43de-95f0-4d939b864aac-istiod-ca-cert\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.249428 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.249315 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-data\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.249428 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.249346 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpd4c\" (UniqueName: \"kubernetes.io/projected/8fd72fbc-016a-43de-95f0-4d939b864aac-kube-api-access-cpd4c\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.249428 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.249417 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-workload-certs\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.249578 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.249489 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-credential-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.249809 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.249786 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-workload-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.250007 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.249990 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd72fbc-016a-43de-95f0-4d939b864aac-istiod-ca-cert\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.251901 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.251876 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-envoy\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.252061 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.252042 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-podinfo\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.258009 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.257987 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-token\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.258127 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.258073 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpd4c\" (UniqueName: \"kubernetes.io/projected/8fd72fbc-016a-43de-95f0-4d939b864aac-kube-api-access-cpd4c\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.445645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.445584 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:43.739269 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:08:43.739225 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd72fbc_016a_43de_95f0_4d939b864aac.slice/crio-1a78c2b521ddb62cf496943013605dec09fe2e4142c781d6a159f4e8a4f649dd WatchSource:0}: Error finding container 1a78c2b521ddb62cf496943013605dec09fe2e4142c781d6a159f4e8a4f649dd: Status 404 returned error can't find the container with id 1a78c2b521ddb62cf496943013605dec09fe2e4142c781d6a159f4e8a4f649dd Nov 02 18:08:43.739969 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.739942 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc"] Nov 02 18:08:43.742542 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.742510 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:08:43.742690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.742574 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:08:43.742690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:43.742607 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:08:44.085857 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:44.085779 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" event={"ID":"8fd72fbc-016a-43de-95f0-4d939b864aac","Type":"ContainerStarted","Data":"1cd6d046ceb47e2e820dcd8d41d0bdc2f247a635962d75f60256b950e1b09ff5"} Nov 02 18:08:44.085857 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:44.085829 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" event={"ID":"8fd72fbc-016a-43de-95f0-4d939b864aac","Type":"ContainerStarted","Data":"1a78c2b521ddb62cf496943013605dec09fe2e4142c781d6a159f4e8a4f649dd"} Nov 02 18:08:44.109782 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:44.109731 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" podStartSLOduration=1.109716602 podStartE2EDuration="1.109716602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:44.107244218 +0000 UTC m=+2088.798298232" watchObservedRunningTime="2026-11-02 18:08:44.109716602 +0000 UTC m=+2088.800770613" Nov 02 18:08:44.455082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:44.448985 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:44.464642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:44.460762 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 02 18:08:44.464642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:44.460829 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" podUID="8fd72fbc-016a-43de-95f0-4d939b864aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 02 18:08:45.459159 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:45.459115 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 02 18:08:45.459577 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:45.459182 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" podUID="8fd72fbc-016a-43de-95f0-4d939b864aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 02 18:08:46.458694 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:46.448505 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 02 18:08:46.458694 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:46.448568 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" podUID="8fd72fbc-016a-43de-95f0-4d939b864aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 02 18:08:47.458643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:47.458097 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 02 18:08:47.458643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:47.458177 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" podUID="8fd72fbc-016a-43de-95f0-4d939b864aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 02 18:08:48.458764 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:48.458715 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 02 18:08:48.459261 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:48.458782 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" podUID="8fd72fbc-016a-43de-95f0-4d939b864aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 02 18:08:49.455641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:49.451376 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 02 18:08:49.455641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:49.451441 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" podUID="8fd72fbc-016a-43de-95f0-4d939b864aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 02 18:08:50.451098 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:50.451068 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:50.451587 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:50.451559 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:50.452351 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:50.452326 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:08:51.887893 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:51.887851 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78"] Nov 02 18:08:53.964710 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:53.964284 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8"] Nov 02 18:08:56.888758 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:56.888670 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" podUID="1b831430-b7e7-4e83-a59a-ebc46dcc8002" containerName="payload-processing" containerID="cri-o://ebefb05e65d191ef5262830dfda3398357d9636b77184a449c9d80f0b274a1fe" gracePeriod=25 Nov 02 18:08:57.173808 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.173783 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" Nov 02 18:08:57.203811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.203197 5601 generic.go:356] "Generic (PLEG): container finished" podID="1b831430-b7e7-4e83-a59a-ebc46dcc8002" containerID="ebefb05e65d191ef5262830dfda3398357d9636b77184a449c9d80f0b274a1fe" exitCode=0 Nov 02 18:08:57.203811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.203274 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" Nov 02 18:08:57.203811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.203429 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" event={"ID":"1b831430-b7e7-4e83-a59a-ebc46dcc8002","Type":"ContainerDied","Data":"ebefb05e65d191ef5262830dfda3398357d9636b77184a449c9d80f0b274a1fe"} Nov 02 18:08:57.203811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.203475 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78" event={"ID":"1b831430-b7e7-4e83-a59a-ebc46dcc8002","Type":"ContainerDied","Data":"981c791f44c2a8e0833f05f8aadbb6a3f0eb37e8c2c7623d7f701172c98fff6e"} Nov 02 18:08:57.203811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.203507 5601 scope.go:122] "RemoveContainer" containerID="ebefb05e65d191ef5262830dfda3398357d9636b77184a449c9d80f0b274a1fe" Nov 02 18:08:57.218521 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.218422 5601 scope.go:122] "RemoveContainer" containerID="ebefb05e65d191ef5262830dfda3398357d9636b77184a449c9d80f0b274a1fe" Nov 02 18:08:57.218805 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:08:57.218781 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebefb05e65d191ef5262830dfda3398357d9636b77184a449c9d80f0b274a1fe\": container with ID starting with ebefb05e65d191ef5262830dfda3398357d9636b77184a449c9d80f0b274a1fe not found: ID does not exist" containerID="ebefb05e65d191ef5262830dfda3398357d9636b77184a449c9d80f0b274a1fe" Nov 02 18:08:57.218928 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.218812 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebefb05e65d191ef5262830dfda3398357d9636b77184a449c9d80f0b274a1fe"} err="failed to get container status \"ebefb05e65d191ef5262830dfda3398357d9636b77184a449c9d80f0b274a1fe\": rpc error: code = NotFound desc = could not find container \"ebefb05e65d191ef5262830dfda3398357d9636b77184a449c9d80f0b274a1fe\": container with ID starting with ebefb05e65d191ef5262830dfda3398357d9636b77184a449c9d80f0b274a1fe not found: ID does not exist" Nov 02 18:08:57.249808 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.249772 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b831430-b7e7-4e83-a59a-ebc46dcc8002-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b831430-b7e7-4e83-a59a-ebc46dcc8002-provider-credentials\") pod \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\" (UID: \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\") " Nov 02 18:08:57.249948 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.249865 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b831430-b7e7-4e83-a59a-ebc46dcc8002-kube-api-access-gmjxw\" (UniqueName: \"kubernetes.io/projected/1b831430-b7e7-4e83-a59a-ebc46dcc8002-kube-api-access-gmjxw\") pod \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\" (UID: \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\") " Nov 02 18:08:57.250092 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.250075 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b831430-b7e7-4e83-a59a-ebc46dcc8002-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b831430-b7e7-4e83-a59a-ebc46dcc8002-plugins-config-volume\") pod \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\" (UID: \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\") " Nov 02 18:08:57.250189 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.250176 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b831430-b7e7-4e83-a59a-ebc46dcc8002-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b831430-b7e7-4e83-a59a-ebc46dcc8002-routing-overlay-volume\") pod \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\" (UID: \"1b831430-b7e7-4e83-a59a-ebc46dcc8002\") " Nov 02 18:08:57.251096 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.250683 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b831430-b7e7-4e83-a59a-ebc46dcc8002-plugins-config-volume" pod "1b831430-b7e7-4e83-a59a-ebc46dcc8002" (UID: "1b831430-b7e7-4e83-a59a-ebc46dcc8002"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:08:57.251096 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.251049 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b831430-b7e7-4e83-a59a-ebc46dcc8002-routing-overlay-volume" pod "1b831430-b7e7-4e83-a59a-ebc46dcc8002" (UID: "1b831430-b7e7-4e83-a59a-ebc46dcc8002"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:08:57.251247 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.251213 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b831430-b7e7-4e83-a59a-ebc46dcc8002-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:57.251247 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.251233 5601 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b831430-b7e7-4e83-a59a-ebc46dcc8002-routing-overlay-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:57.252884 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.252847 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b831430-b7e7-4e83-a59a-ebc46dcc8002-provider-credentials" pod "1b831430-b7e7-4e83-a59a-ebc46dcc8002" (UID: "1b831430-b7e7-4e83-a59a-ebc46dcc8002"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:57.252988 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.252883 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b831430-b7e7-4e83-a59a-ebc46dcc8002-kube-api-access-gmjxw" pod "1b831430-b7e7-4e83-a59a-ebc46dcc8002" (UID: "1b831430-b7e7-4e83-a59a-ebc46dcc8002"). InnerVolumeSpecName "kube-api-access-gmjxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:57.353161 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.353129 5601 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b831430-b7e7-4e83-a59a-ebc46dcc8002-provider-credentials\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:57.353161 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.353158 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmjxw\" (UniqueName: \"kubernetes.io/projected/1b831430-b7e7-4e83-a59a-ebc46dcc8002-kube-api-access-gmjxw\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:57.537250 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.537213 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78"] Nov 02 18:08:57.541748 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:57.541720 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-5dcdbrk78"] Nov 02 18:08:58.965791 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:58.965656 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" podUID="0ac163f9-7481-4312-9c8b-ffdebfd383b6" containerName="payload-processing" containerID="cri-o://d83ad43f7dbe4bba304320575f0b9427d58dffd3a7a12a97c79bbd3c77ffc07d" gracePeriod=25 Nov 02 18:08:59.225517 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:59.225416 5601 generic.go:356] "Generic (PLEG): container finished" podID="0ac163f9-7481-4312-9c8b-ffdebfd383b6" containerID="d83ad43f7dbe4bba304320575f0b9427d58dffd3a7a12a97c79bbd3c77ffc07d" exitCode=0 Nov 02 18:08:59.225517 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:59.225493 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" event={"ID":"0ac163f9-7481-4312-9c8b-ffdebfd383b6","Type":"ContainerDied","Data":"d83ad43f7dbe4bba304320575f0b9427d58dffd3a7a12a97c79bbd3c77ffc07d"} Nov 02 18:08:59.277244 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:59.277218 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" Nov 02 18:08:59.413261 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:59.413223 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ac163f9-7481-4312-9c8b-ffdebfd383b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac163f9-7481-4312-9c8b-ffdebfd383b6-plugins-config-volume\") pod \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\" (UID: \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\") " Nov 02 18:08:59.413453 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:59.413285 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ac163f9-7481-4312-9c8b-ffdebfd383b6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ac163f9-7481-4312-9c8b-ffdebfd383b6-routing-overlay-volume\") pod \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\" (UID: \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\") " Nov 02 18:08:59.413527 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:59.413478 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac163f9-7481-4312-9c8b-ffdebfd383b6-kube-api-access-hgtn6\" (UniqueName: \"kubernetes.io/projected/0ac163f9-7481-4312-9c8b-ffdebfd383b6-kube-api-access-hgtn6\") pod \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\" (UID: \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\") " Nov 02 18:08:59.413527 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:59.413519 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac163f9-7481-4312-9c8b-ffdebfd383b6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ac163f9-7481-4312-9c8b-ffdebfd383b6-provider-credentials\") pod \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\" (UID: \"0ac163f9-7481-4312-9c8b-ffdebfd383b6\") " Nov 02 18:08:59.414430 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:59.414400 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac163f9-7481-4312-9c8b-ffdebfd383b6-routing-overlay-volume" pod "0ac163f9-7481-4312-9c8b-ffdebfd383b6" (UID: "0ac163f9-7481-4312-9c8b-ffdebfd383b6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:08:59.415030 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:59.414994 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac163f9-7481-4312-9c8b-ffdebfd383b6-plugins-config-volume" pod "0ac163f9-7481-4312-9c8b-ffdebfd383b6" (UID: "0ac163f9-7481-4312-9c8b-ffdebfd383b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:08:59.415891 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:59.415866 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac163f9-7481-4312-9c8b-ffdebfd383b6-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:59.415965 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:59.415894 5601 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ac163f9-7481-4312-9c8b-ffdebfd383b6-routing-overlay-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:59.417100 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:59.417075 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac163f9-7481-4312-9c8b-ffdebfd383b6-provider-credentials" pod "0ac163f9-7481-4312-9c8b-ffdebfd383b6" (UID: "0ac163f9-7481-4312-9c8b-ffdebfd383b6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:59.417236 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:59.417214 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac163f9-7481-4312-9c8b-ffdebfd383b6-kube-api-access-hgtn6" pod "0ac163f9-7481-4312-9c8b-ffdebfd383b6" (UID: "0ac163f9-7481-4312-9c8b-ffdebfd383b6"). InnerVolumeSpecName "kube-api-access-hgtn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:59.429729 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:59.429702 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b831430-b7e7-4e83-a59a-ebc46dcc8002" path="/var/lib/kubelet/pods/1b831430-b7e7-4e83-a59a-ebc46dcc8002/volumes" Nov 02 18:08:59.517839 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:59.517765 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgtn6\" (UniqueName: \"kubernetes.io/projected/0ac163f9-7481-4312-9c8b-ffdebfd383b6-kube-api-access-hgtn6\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:59.517839 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:08:59.517791 5601 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ac163f9-7481-4312-9c8b-ffdebfd383b6-provider-credentials\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:00.290638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:09:00.290283 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" event={"ID":"0ac163f9-7481-4312-9c8b-ffdebfd383b6","Type":"ContainerDied","Data":"0f4a275b53769f56b1650de4f777f3b8744645ff70a3fadc07a5e54f78b9a34b"} Nov 02 18:09:00.290638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:09:00.290370 5601 scope.go:122] "RemoveContainer" containerID="d83ad43f7dbe4bba304320575f0b9427d58dffd3a7a12a97c79bbd3c77ffc07d" Nov 02 18:09:00.290638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:09:00.290527 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8" Nov 02 18:09:00.327641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:09:00.324531 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8"] Nov 02 18:09:00.347009 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:09:00.338049 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-77552fvv8"] Nov 02 18:09:01.428416 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:09:01.428375 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ac163f9-7481-4312-9c8b-ffdebfd383b6" path="/var/lib/kubelet/pods/0ac163f9-7481-4312-9c8b-ffdebfd383b6/volumes" Nov 02 18:09:04.487216 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:09:04.487171 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:09:34.655956 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:09:34.655877 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:10:00.433389 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:00.433339 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6"] Nov 02 18:10:01.819603 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:01.819552 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-jv8f4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:10:01.831773 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:01.831726 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" podUID="c32fa6f3-9665-4b3b-b093-22435190829b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:10:02.037642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:02.033740 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-wtks5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:10:02.037642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:02.033868 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" podUID="63f70366-70f1-475a-8e45-eb6da6b4828e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:10:02.775638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:02.770738 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-mdxxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:10:02.775638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:02.770842 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" podUID="2f57ca2f-9916-45aa-9523-65e8fd925f15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:10:04.487886 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:10:04.487848 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:10:05.475328 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:05.475282 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" podUID="0db86edb-4352-4d70-a99f-4070d736fb52" containerName="payload-processing" containerID="cri-o://5e4b383778e99f7f9db3aab736a7c358534139d0c0a8f18a2797a6dabd21fec1" gracePeriod=25 Nov 02 18:10:05.860053 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:05.860024 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" Nov 02 18:10:06.029279 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.029221 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0db86edb-4352-4d70-a99f-4070d736fb52-kube-api-access-zz5vp\" (UniqueName: \"kubernetes.io/projected/0db86edb-4352-4d70-a99f-4070d736fb52-kube-api-access-zz5vp\") pod \"0db86edb-4352-4d70-a99f-4070d736fb52\" (UID: \"0db86edb-4352-4d70-a99f-4070d736fb52\") " Nov 02 18:10:06.029468 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.029373 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0db86edb-4352-4d70-a99f-4070d736fb52-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0db86edb-4352-4d70-a99f-4070d736fb52-provider-credentials\") pod \"0db86edb-4352-4d70-a99f-4070d736fb52\" (UID: \"0db86edb-4352-4d70-a99f-4070d736fb52\") " Nov 02 18:10:06.029468 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.029426 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0db86edb-4352-4d70-a99f-4070d736fb52-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0db86edb-4352-4d70-a99f-4070d736fb52-routing-overlay-volume\") pod \"0db86edb-4352-4d70-a99f-4070d736fb52\" (UID: \"0db86edb-4352-4d70-a99f-4070d736fb52\") " Nov 02 18:10:06.029663 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.029631 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0db86edb-4352-4d70-a99f-4070d736fb52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db86edb-4352-4d70-a99f-4070d736fb52-plugins-config-volume\") pod \"0db86edb-4352-4d70-a99f-4070d736fb52\" (UID: \"0db86edb-4352-4d70-a99f-4070d736fb52\") " Nov 02 18:10:06.029925 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.029898 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db86edb-4352-4d70-a99f-4070d736fb52-routing-overlay-volume" pod "0db86edb-4352-4d70-a99f-4070d736fb52" (UID: "0db86edb-4352-4d70-a99f-4070d736fb52"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:06.030547 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.030487 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db86edb-4352-4d70-a99f-4070d736fb52-plugins-config-volume" pod "0db86edb-4352-4d70-a99f-4070d736fb52" (UID: "0db86edb-4352-4d70-a99f-4070d736fb52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:06.030547 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.030498 5601 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0db86edb-4352-4d70-a99f-4070d736fb52-routing-overlay-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:06.032243 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.032214 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db86edb-4352-4d70-a99f-4070d736fb52-provider-credentials" pod "0db86edb-4352-4d70-a99f-4070d736fb52" (UID: "0db86edb-4352-4d70-a99f-4070d736fb52"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:06.032370 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.032342 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db86edb-4352-4d70-a99f-4070d736fb52-kube-api-access-zz5vp" pod "0db86edb-4352-4d70-a99f-4070d736fb52" (UID: "0db86edb-4352-4d70-a99f-4070d736fb52"). InnerVolumeSpecName "kube-api-access-zz5vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:06.131886 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.131808 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz5vp\" (UniqueName: \"kubernetes.io/projected/0db86edb-4352-4d70-a99f-4070d736fb52-kube-api-access-zz5vp\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:06.131886 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.131838 5601 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0db86edb-4352-4d70-a99f-4070d736fb52-provider-credentials\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:06.131886 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.131849 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db86edb-4352-4d70-a99f-4070d736fb52-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:06.368473 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.368439 5601 generic.go:356] "Generic (PLEG): container finished" podID="0db86edb-4352-4d70-a99f-4070d736fb52" containerID="5e4b383778e99f7f9db3aab736a7c358534139d0c0a8f18a2797a6dabd21fec1" exitCode=0 Nov 02 18:10:06.368776 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.368493 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" Nov 02 18:10:06.368776 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.368518 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" event={"ID":"0db86edb-4352-4d70-a99f-4070d736fb52","Type":"ContainerDied","Data":"5e4b383778e99f7f9db3aab736a7c358534139d0c0a8f18a2797a6dabd21fec1"} Nov 02 18:10:06.368776 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.368577 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6" event={"ID":"0db86edb-4352-4d70-a99f-4070d736fb52","Type":"ContainerDied","Data":"5cf3da93a97ce0af3f8f8ba35e1e4da3ece123ad6ff374d022b555b80f35a24c"} Nov 02 18:10:06.368776 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.368603 5601 scope.go:122] "RemoveContainer" containerID="5e4b383778e99f7f9db3aab736a7c358534139d0c0a8f18a2797a6dabd21fec1" Nov 02 18:10:06.387018 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.386988 5601 scope.go:122] "RemoveContainer" containerID="5e4b383778e99f7f9db3aab736a7c358534139d0c0a8f18a2797a6dabd21fec1" Nov 02 18:10:06.387345 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:10:06.387328 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e4b383778e99f7f9db3aab736a7c358534139d0c0a8f18a2797a6dabd21fec1\": container with ID starting with 5e4b383778e99f7f9db3aab736a7c358534139d0c0a8f18a2797a6dabd21fec1 not found: ID does not exist" containerID="5e4b383778e99f7f9db3aab736a7c358534139d0c0a8f18a2797a6dabd21fec1" Nov 02 18:10:06.387387 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.387354 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e4b383778e99f7f9db3aab736a7c358534139d0c0a8f18a2797a6dabd21fec1"} err="failed to get container status \"5e4b383778e99f7f9db3aab736a7c358534139d0c0a8f18a2797a6dabd21fec1\": rpc error: code = NotFound desc = could not find container \"5e4b383778e99f7f9db3aab736a7c358534139d0c0a8f18a2797a6dabd21fec1\": container with ID starting with 5e4b383778e99f7f9db3aab736a7c358534139d0c0a8f18a2797a6dabd21fec1 not found: ID does not exist" Nov 02 18:10:06.399279 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.399239 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6"] Nov 02 18:10:06.410245 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:06.410203 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-5ec436/payload-processing-external-model-e2e-extmod-a-5ec436-7f97z26w6"] Nov 02 18:10:07.431345 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:07.431254 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0db86edb-4352-4d70-a99f-4070d736fb52" path="/var/lib/kubelet/pods/0db86edb-4352-4d70-a99f-4070d736fb52/volumes" Nov 02 18:10:34.490804 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:10:34.490758 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:10:53.409606 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:53.409566 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2"] Nov 02 18:10:53.410160 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:53.409788 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" podUID="a268dccb-3642-470f-b0b2-d9e4c4257ca5" containerName="istio-proxy" containerID="cri-o://d86f80539de6b2f21a64b823dd2c3ef3e7d30ebd7c698969905b7eb394f46ecf" gracePeriod=30 Nov 02 18:10:58.789466 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.789432 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 18:10:58.953390 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.953359 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-workload-socket\") pod \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " Nov 02 18:10:58.953573 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.953396 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-workload-certs\") pod \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " Nov 02 18:10:58.953573 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.953428 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a268dccb-3642-470f-b0b2-d9e4c4257ca5-kube-api-access-zkk5x\" (UniqueName: \"kubernetes.io/projected/a268dccb-3642-470f-b0b2-d9e4c4257ca5-kube-api-access-zkk5x\") pod \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " Nov 02 18:10:58.953573 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.953510 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-podinfo\") pod \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " Nov 02 18:10:58.953822 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.953595 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-data\") pod \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " Nov 02 18:10:58.953822 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.953709 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-envoy\") pod \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " Nov 02 18:10:58.953822 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.953758 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-workload-certs" pod "a268dccb-3642-470f-b0b2-d9e4c4257ca5" (UID: "a268dccb-3642-470f-b0b2-d9e4c4257ca5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:58.953822 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.953770 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-workload-socket" pod "a268dccb-3642-470f-b0b2-d9e4c4257ca5" (UID: "a268dccb-3642-470f-b0b2-d9e4c4257ca5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:58.953822 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.953776 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-token\" (UniqueName: \"kubernetes.io/projected/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-token\") pod \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " Nov 02 18:10:58.954094 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.953859 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-data" pod "a268dccb-3642-470f-b0b2-d9e4c4257ca5" (UID: "a268dccb-3642-470f-b0b2-d9e4c4257ca5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:58.954536 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.954509 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istiod-ca-cert\") pod \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " Nov 02 18:10:58.954705 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.954585 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-credential-socket\") pod \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\" (UID: \"a268dccb-3642-470f-b0b2-d9e4c4257ca5\") " Nov 02 18:10:58.955343 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.954979 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-credential-socket" pod "a268dccb-3642-470f-b0b2-d9e4c4257ca5" (UID: "a268dccb-3642-470f-b0b2-d9e4c4257ca5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:58.955802 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.955563 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:58.955802 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.955588 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:58.955802 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.955604 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:58.955802 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.955651 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:58.957906 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.955964 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istiod-ca-cert" pod "a268dccb-3642-470f-b0b2-d9e4c4257ca5" (UID: "a268dccb-3642-470f-b0b2-d9e4c4257ca5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:58.957906 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.956342 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-envoy" pod "a268dccb-3642-470f-b0b2-d9e4c4257ca5" (UID: "a268dccb-3642-470f-b0b2-d9e4c4257ca5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:58.957906 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.956457 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-token" pod "a268dccb-3642-470f-b0b2-d9e4c4257ca5" (UID: "a268dccb-3642-470f-b0b2-d9e4c4257ca5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:58.957906 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.956849 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a268dccb-3642-470f-b0b2-d9e4c4257ca5-kube-api-access-zkk5x" pod "a268dccb-3642-470f-b0b2-d9e4c4257ca5" (UID: "a268dccb-3642-470f-b0b2-d9e4c4257ca5"). InnerVolumeSpecName "kube-api-access-zkk5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:58.958180 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:58.958031 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-podinfo" pod "a268dccb-3642-470f-b0b2-d9e4c4257ca5" (UID: "a268dccb-3642-470f-b0b2-d9e4c4257ca5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:10:59.057992 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:59.057907 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkk5x\" (UniqueName: \"kubernetes.io/projected/a268dccb-3642-470f-b0b2-d9e4c4257ca5-kube-api-access-zkk5x\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:59.057992 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:59.057947 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:59.057992 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:59.057965 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:59.057992 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:59.057979 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:59.057992 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:59.057995 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a268dccb-3642-470f-b0b2-d9e4c4257ca5-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:10:59.234257 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:59.234220 5601 generic.go:356] "Generic (PLEG): container finished" podID="a268dccb-3642-470f-b0b2-d9e4c4257ca5" containerID="d86f80539de6b2f21a64b823dd2c3ef3e7d30ebd7c698969905b7eb394f46ecf" exitCode=0 Nov 02 18:10:59.234476 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:59.234268 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" Nov 02 18:10:59.234476 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:59.234303 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" event={"ID":"a268dccb-3642-470f-b0b2-d9e4c4257ca5","Type":"ContainerDied","Data":"d86f80539de6b2f21a64b823dd2c3ef3e7d30ebd7c698969905b7eb394f46ecf"} Nov 02 18:10:59.234476 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:59.234358 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2" event={"ID":"a268dccb-3642-470f-b0b2-d9e4c4257ca5","Type":"ContainerDied","Data":"ec4394472b6598c599ab16c634e4d39fc2acccec89b21b08f1977d116d4c3cce"} Nov 02 18:10:59.234476 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:59.234380 5601 scope.go:122] "RemoveContainer" containerID="d86f80539de6b2f21a64b823dd2c3ef3e7d30ebd7c698969905b7eb394f46ecf" Nov 02 18:10:59.246989 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:59.246972 5601 scope.go:122] "RemoveContainer" containerID="d86f80539de6b2f21a64b823dd2c3ef3e7d30ebd7c698969905b7eb394f46ecf" Nov 02 18:10:59.247262 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:10:59.247243 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d86f80539de6b2f21a64b823dd2c3ef3e7d30ebd7c698969905b7eb394f46ecf\": container with ID starting with d86f80539de6b2f21a64b823dd2c3ef3e7d30ebd7c698969905b7eb394f46ecf not found: ID does not exist" containerID="d86f80539de6b2f21a64b823dd2c3ef3e7d30ebd7c698969905b7eb394f46ecf" Nov 02 18:10:59.247312 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:59.247269 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d86f80539de6b2f21a64b823dd2c3ef3e7d30ebd7c698969905b7eb394f46ecf"} err="failed to get container status \"d86f80539de6b2f21a64b823dd2c3ef3e7d30ebd7c698969905b7eb394f46ecf\": rpc error: code = NotFound desc = could not find container \"d86f80539de6b2f21a64b823dd2c3ef3e7d30ebd7c698969905b7eb394f46ecf\": container with ID starting with d86f80539de6b2f21a64b823dd2c3ef3e7d30ebd7c698969905b7eb394f46ecf not found: ID does not exist" Nov 02 18:10:59.262251 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:59.262223 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2"] Nov 02 18:10:59.268092 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:59.268067 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-7llc2"] Nov 02 18:10:59.428745 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:10:59.428714 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a268dccb-3642-470f-b0b2-d9e4c4257ca5" path="/var/lib/kubelet/pods/a268dccb-3642-470f-b0b2-d9e4c4257ca5/volumes" Nov 02 18:11:02.904857 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.904824 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2"] Nov 02 18:11:02.905560 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.905544 5601 cpu_manager.go:414] "containerMap: removing container" podUID="0ac163f9-7481-4312-9c8b-ffdebfd383b6" containerName="payload-processing" Nov 02 18:11:02.905652 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.905561 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ac163f9-7481-4312-9c8b-ffdebfd383b6" containerName="payload-processing" Nov 02 18:11:02.905652 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.905595 5601 cpu_manager.go:414] "containerMap: removing container" podUID="a268dccb-3642-470f-b0b2-d9e4c4257ca5" containerName="istio-proxy" Nov 02 18:11:02.905652 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.905600 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a268dccb-3642-470f-b0b2-d9e4c4257ca5" containerName="istio-proxy" Nov 02 18:11:02.905652 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.905648 5601 cpu_manager.go:414] "containerMap: removing container" podUID="1b831430-b7e7-4e83-a59a-ebc46dcc8002" containerName="payload-processing" Nov 02 18:11:02.905652 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.905655 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b831430-b7e7-4e83-a59a-ebc46dcc8002" containerName="payload-processing" Nov 02 18:11:02.905823 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.905668 5601 cpu_manager.go:414] "containerMap: removing container" podUID="0db86edb-4352-4d70-a99f-4070d736fb52" containerName="payload-processing" Nov 02 18:11:02.905823 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.905675 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0db86edb-4352-4d70-a99f-4070d736fb52" containerName="payload-processing" Nov 02 18:11:02.905892 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.905855 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="a268dccb-3642-470f-b0b2-d9e4c4257ca5" containerName="istio-proxy" Nov 02 18:11:02.905892 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.905879 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b831430-b7e7-4e83-a59a-ebc46dcc8002" containerName="payload-processing" Nov 02 18:11:02.905892 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.905889 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="0db86edb-4352-4d70-a99f-4070d736fb52" containerName="payload-processing" Nov 02 18:11:02.905987 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.905900 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ac163f9-7481-4312-9c8b-ffdebfd383b6" containerName="payload-processing" Nov 02 18:11:02.911833 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.911789 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" Nov 02 18:11:02.918217 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.916938 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"kube-root-ca.crt\"" Nov 02 18:11:02.918217 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.916952 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"agc-cred-735222da8e08\"" Nov 02 18:11:02.918217 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.916938 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"routing-overlay\"" Nov 02 18:11:02.918217 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.916938 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"openshift-service-ca.crt\"" Nov 02 18:11:02.918217 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.916970 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"payload-processing-external-model-plugins-e2e-extmod-a-807c08\"" Nov 02 18:11:02.918217 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.916994 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"agc-cred-41558d72238a\"" Nov 02 18:11:02.918217 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.917133 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"payload-processing-external-model-e2e--d2178d1f-dockercfg-8gtsz\"" Nov 02 18:11:02.918217 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.917287 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2"] Nov 02 18:11:02.963929 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.963889 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a525fd4-e6e8-4ffa-a902-b89637d8d993-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2\" (UID: \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" Nov 02 18:11:02.964107 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.964009 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-252f2\" (UniqueName: \"kubernetes.io/projected/1a525fd4-e6e8-4ffa-a902-b89637d8d993-kube-api-access-252f2\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2\" (UID: \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" Nov 02 18:11:02.964180 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.964106 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a525fd4-e6e8-4ffa-a902-b89637d8d993-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2\" (UID: \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" Nov 02 18:11:02.964180 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:02.964147 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a525fd4-e6e8-4ffa-a902-b89637d8d993-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2\" (UID: \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" Nov 02 18:11:03.066056 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.066019 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a525fd4-e6e8-4ffa-a902-b89637d8d993-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2\" (UID: \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" Nov 02 18:11:03.066246 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.066078 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a525fd4-e6e8-4ffa-a902-b89637d8d993-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2\" (UID: \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" Nov 02 18:11:03.066307 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.066283 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a525fd4-e6e8-4ffa-a902-b89637d8d993-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2\" (UID: \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" Nov 02 18:11:03.066351 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.066339 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-252f2\" (UniqueName: \"kubernetes.io/projected/1a525fd4-e6e8-4ffa-a902-b89637d8d993-kube-api-access-252f2\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2\" (UID: \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" Nov 02 18:11:03.066914 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.066886 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a525fd4-e6e8-4ffa-a902-b89637d8d993-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2\" (UID: \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" Nov 02 18:11:03.066914 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.066902 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a525fd4-e6e8-4ffa-a902-b89637d8d993-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2\" (UID: \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" Nov 02 18:11:03.069485 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.069456 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a525fd4-e6e8-4ffa-a902-b89637d8d993-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2\" (UID: \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" Nov 02 18:11:03.074779 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.074748 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-252f2\" (UniqueName: \"kubernetes.io/projected/1a525fd4-e6e8-4ffa-a902-b89637d8d993-kube-api-access-252f2\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2\" (UID: \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" Nov 02 18:11:03.196756 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.196717 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl"] Nov 02 18:11:03.201720 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.201695 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" Nov 02 18:11:03.207496 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.207461 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"routing-overlay\"" Nov 02 18:11:03.208042 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.207989 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl"] Nov 02 18:11:03.208203 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.208134 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"kube-root-ca.crt\"" Nov 02 18:11:03.208203 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.208165 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"openshift-service-ca.crt\"" Nov 02 18:11:03.208673 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.208601 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"agc-cred-5df426b52b49\"" Nov 02 18:11:03.208673 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.208641 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"agc-cred-58b28beb62f9\"" Nov 02 18:11:03.208840 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.208772 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"payload-processing-external-model-e2e--24159a0a-dockercfg-lk82g\"" Nov 02 18:11:03.215422 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.215401 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"payload-processing-external-model-plugins-e2e-extmod-b-bee8f6\"" Nov 02 18:11:03.231932 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.231631 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" Nov 02 18:11:03.270333 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.270289 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46c9a1a3-ff28-4d6c-9d85-faa3be401911-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl\" (UID: \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" Nov 02 18:11:03.270744 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.270712 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46c9a1a3-ff28-4d6c-9d85-faa3be401911-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl\" (UID: \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" Nov 02 18:11:03.270829 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.270783 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c9a1a3-ff28-4d6c-9d85-faa3be401911-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl\" (UID: \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" Nov 02 18:11:03.270958 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.270932 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt6wr\" (UniqueName: \"kubernetes.io/projected/46c9a1a3-ff28-4d6c-9d85-faa3be401911-kube-api-access-jt6wr\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl\" (UID: \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" Nov 02 18:11:03.372901 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.372857 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46c9a1a3-ff28-4d6c-9d85-faa3be401911-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl\" (UID: \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" Nov 02 18:11:03.373101 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.372934 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c9a1a3-ff28-4d6c-9d85-faa3be401911-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl\" (UID: \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" Nov 02 18:11:03.373101 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.373057 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt6wr\" (UniqueName: \"kubernetes.io/projected/46c9a1a3-ff28-4d6c-9d85-faa3be401911-kube-api-access-jt6wr\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl\" (UID: \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" Nov 02 18:11:03.373231 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.373120 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46c9a1a3-ff28-4d6c-9d85-faa3be401911-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl\" (UID: \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" Nov 02 18:11:03.373821 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.373795 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c9a1a3-ff28-4d6c-9d85-faa3be401911-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl\" (UID: \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" Nov 02 18:11:03.373959 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.373908 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46c9a1a3-ff28-4d6c-9d85-faa3be401911-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl\" (UID: \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" Nov 02 18:11:03.375591 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.375561 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46c9a1a3-ff28-4d6c-9d85-faa3be401911-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl\" (UID: \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" Nov 02 18:11:03.383041 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.383015 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt6wr\" (UniqueName: \"kubernetes.io/projected/46c9a1a3-ff28-4d6c-9d85-faa3be401911-kube-api-access-jt6wr\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl\" (UID: \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" Nov 02 18:11:03.544752 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.544660 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" Nov 02 18:11:03.557895 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.557866 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2"] Nov 02 18:11:03.561312 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:11:03.561278 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a525fd4_e6e8_4ffa_a902_b89637d8d993.slice/crio-0277bc133e2d453d0470c423d03982b6d05661581a2ed3a5598cf0e28dc07300 WatchSource:0}: Error finding container 0277bc133e2d453d0470c423d03982b6d05661581a2ed3a5598cf0e28dc07300: Status 404 returned error can't find the container with id 0277bc133e2d453d0470c423d03982b6d05661581a2ed3a5598cf0e28dc07300 Nov 02 18:11:03.856718 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:03.856681 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl"] Nov 02 18:11:03.857155 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:11:03.856811 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c9a1a3_ff28_4d6c_9d85_faa3be401911.slice/crio-dd9f506b238a4a30ab6f7b8e3bbf70cdf4c03349e9e118315d195d54e0b062f1 WatchSource:0}: Error finding container dd9f506b238a4a30ab6f7b8e3bbf70cdf4c03349e9e118315d195d54e0b062f1: Status 404 returned error can't find the container with id dd9f506b238a4a30ab6f7b8e3bbf70cdf4c03349e9e118315d195d54e0b062f1 Nov 02 18:11:04.317330 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:04.317290 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" event={"ID":"46c9a1a3-ff28-4d6c-9d85-faa3be401911","Type":"ContainerStarted","Data":"9c9cf3b65b791c23410e500f1ac0313ab7bc27bf3d751b480fbeee12de5bc7fd"} Nov 02 18:11:04.317872 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:04.317353 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" event={"ID":"46c9a1a3-ff28-4d6c-9d85-faa3be401911","Type":"ContainerStarted","Data":"dd9f506b238a4a30ab6f7b8e3bbf70cdf4c03349e9e118315d195d54e0b062f1"} Nov 02 18:11:04.317872 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:04.317379 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" Nov 02 18:11:04.319333 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:04.319301 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" event={"ID":"1a525fd4-e6e8-4ffa-a902-b89637d8d993","Type":"ContainerStarted","Data":"2dc0a6f6d733eb0bbe322a9f2085e2879254a8a77016417ef46ed709ea952152"} Nov 02 18:11:04.319445 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:04.319360 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" event={"ID":"1a525fd4-e6e8-4ffa-a902-b89637d8d993","Type":"ContainerStarted","Data":"0277bc133e2d453d0470c423d03982b6d05661581a2ed3a5598cf0e28dc07300"} Nov 02 18:11:04.341523 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:04.341464 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" podStartSLOduration=1.341445369 podStartE2EDuration="1.341445369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:04.335145019 +0000 UTC m=+2229.026199032" watchObservedRunningTime="2026-11-02 18:11:04.341445369 +0000 UTC m=+2229.032499383" Nov 02 18:11:04.363944 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:04.363893 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" podStartSLOduration=2.363879215 podStartE2EDuration="2.363879215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:04.361711936 +0000 UTC m=+2229.052765949" watchObservedRunningTime="2026-11-02 18:11:04.363879215 +0000 UTC m=+2229.054933269" Nov 02 18:11:04.488708 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:11:04.488656 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:11:05.329578 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:05.329514 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" Nov 02 18:11:15.333027 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:15.332995 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" Nov 02 18:11:16.374892 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:16.374861 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" Nov 02 18:11:34.487482 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:11:34.487441 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:11:37.519568 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.519492 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5"] Nov 02 18:11:37.524913 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.524892 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" Nov 02 18:11:37.531279 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.531250 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5"] Nov 02 18:11:37.532149 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.532125 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"payload-processing-external-model-e2e--59b59218-dockercfg-lp4xp\"" Nov 02 18:11:37.533152 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.533130 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"agc-cred-0ebc5b098a14\"" Nov 02 18:11:37.533283 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.533130 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"kube-root-ca.crt\"" Nov 02 18:11:37.533283 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.533159 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"payload-processing-external-model-plugins-e2e-extmod-a-36a991\"" Nov 02 18:11:37.533283 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.533175 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"agc-cred-72130ce5ee27\"" Nov 02 18:11:37.533283 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.533140 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"agc-cred-811d350f868d\"" Nov 02 18:11:37.533283 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.533226 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"routing-overlay\"" Nov 02 18:11:37.533283 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.533264 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"openshift-service-ca.crt\"" Nov 02 18:11:37.674895 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.674858 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82593a64-c25f-4d0b-8e6b-78816f40984c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5\" (UID: \"82593a64-c25f-4d0b-8e6b-78816f40984c\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" Nov 02 18:11:37.675095 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.674901 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wqjf\" (UniqueName: \"kubernetes.io/projected/82593a64-c25f-4d0b-8e6b-78816f40984c-kube-api-access-6wqjf\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5\" (UID: \"82593a64-c25f-4d0b-8e6b-78816f40984c\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" Nov 02 18:11:37.675095 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.674982 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82593a64-c25f-4d0b-8e6b-78816f40984c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5\" (UID: \"82593a64-c25f-4d0b-8e6b-78816f40984c\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" Nov 02 18:11:37.675095 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.675038 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/82593a64-c25f-4d0b-8e6b-78816f40984c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5\" (UID: \"82593a64-c25f-4d0b-8e6b-78816f40984c\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" Nov 02 18:11:37.776326 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.776236 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82593a64-c25f-4d0b-8e6b-78816f40984c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5\" (UID: \"82593a64-c25f-4d0b-8e6b-78816f40984c\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" Nov 02 18:11:37.776326 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.776278 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wqjf\" (UniqueName: \"kubernetes.io/projected/82593a64-c25f-4d0b-8e6b-78816f40984c-kube-api-access-6wqjf\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5\" (UID: \"82593a64-c25f-4d0b-8e6b-78816f40984c\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" Nov 02 18:11:37.776513 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.776327 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82593a64-c25f-4d0b-8e6b-78816f40984c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5\" (UID: \"82593a64-c25f-4d0b-8e6b-78816f40984c\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" Nov 02 18:11:37.776513 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.776366 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/82593a64-c25f-4d0b-8e6b-78816f40984c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5\" (UID: \"82593a64-c25f-4d0b-8e6b-78816f40984c\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" Nov 02 18:11:37.777033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.777008 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82593a64-c25f-4d0b-8e6b-78816f40984c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5\" (UID: \"82593a64-c25f-4d0b-8e6b-78816f40984c\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" Nov 02 18:11:37.777218 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.777195 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82593a64-c25f-4d0b-8e6b-78816f40984c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5\" (UID: \"82593a64-c25f-4d0b-8e6b-78816f40984c\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" Nov 02 18:11:37.779087 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.779067 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/82593a64-c25f-4d0b-8e6b-78816f40984c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5\" (UID: \"82593a64-c25f-4d0b-8e6b-78816f40984c\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" Nov 02 18:11:37.784346 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.784323 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wqjf\" (UniqueName: \"kubernetes.io/projected/82593a64-c25f-4d0b-8e6b-78816f40984c-kube-api-access-6wqjf\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5\" (UID: \"82593a64-c25f-4d0b-8e6b-78816f40984c\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" Nov 02 18:11:37.843852 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:37.843827 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" Nov 02 18:11:38.130996 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:38.130958 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5"] Nov 02 18:11:38.131328 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:11:38.131299 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82593a64_c25f_4d0b_8e6b_78816f40984c.slice/crio-aa658be7bd4b3738b30229904373e0be2b1a620baaf56f800484ad50de092c5a WatchSource:0}: Error finding container aa658be7bd4b3738b30229904373e0be2b1a620baaf56f800484ad50de092c5a: Status 404 returned error can't find the container with id aa658be7bd4b3738b30229904373e0be2b1a620baaf56f800484ad50de092c5a Nov 02 18:11:38.900070 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:38.899957 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" event={"ID":"82593a64-c25f-4d0b-8e6b-78816f40984c","Type":"ContainerStarted","Data":"956cab20f6cc90b13436973e479532b91a37ad0e56a1b5c45419ff93ae012f02"} Nov 02 18:11:38.900070 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:38.900027 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" event={"ID":"82593a64-c25f-4d0b-8e6b-78816f40984c","Type":"ContainerStarted","Data":"aa658be7bd4b3738b30229904373e0be2b1a620baaf56f800484ad50de092c5a"} Nov 02 18:11:38.901116 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:38.901093 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" Nov 02 18:11:38.923733 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:38.923675 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" podStartSLOduration=1.923656802 podStartE2EDuration="1.923656802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:38.920027822 +0000 UTC m=+2263.611081846" watchObservedRunningTime="2026-11-02 18:11:38.923656802 +0000 UTC m=+2263.614710817" Nov 02 18:11:50.922282 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:11:50.922248 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" Nov 02 18:12:04.489379 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:12:04.489332 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:12:24.571229 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:24.571168 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp"] Nov 02 18:12:24.578014 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:24.571427 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" podUID="5ee859b4-abd7-4add-8511-1d4e2963e182" containerName="istio-proxy" containerID="cri-o://3a69163fac823f5a9926d1fabcac3ccaa3706a44d8c05cef3961ec2be3e0d4be" gracePeriod=30 Nov 02 18:12:29.868330 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.868299 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 18:12:29.962235 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.961053 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-workload-socket\") pod \"5ee859b4-abd7-4add-8511-1d4e2963e182\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " Nov 02 18:12:29.962235 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.961137 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-data\") pod \"5ee859b4-abd7-4add-8511-1d4e2963e182\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " Nov 02 18:12:29.962235 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.961219 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-podinfo\") pod \"5ee859b4-abd7-4add-8511-1d4e2963e182\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " Nov 02 18:12:29.962235 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.961288 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-envoy\") pod \"5ee859b4-abd7-4add-8511-1d4e2963e182\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " Nov 02 18:12:29.962235 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.961342 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-credential-socket\") pod \"5ee859b4-abd7-4add-8511-1d4e2963e182\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " Nov 02 18:12:29.962235 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.961357 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-workload-socket" pod "5ee859b4-abd7-4add-8511-1d4e2963e182" (UID: "5ee859b4-abd7-4add-8511-1d4e2963e182"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:29.962235 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.961549 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-token\" (UniqueName: \"kubernetes.io/projected/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-token\") pod \"5ee859b4-abd7-4add-8511-1d4e2963e182\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " Nov 02 18:12:29.962235 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.961643 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-credential-socket" pod "5ee859b4-abd7-4add-8511-1d4e2963e182" (UID: "5ee859b4-abd7-4add-8511-1d4e2963e182"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:29.962235 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.961730 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee859b4-abd7-4add-8511-1d4e2963e182-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee859b4-abd7-4add-8511-1d4e2963e182-istiod-ca-cert\") pod \"5ee859b4-abd7-4add-8511-1d4e2963e182\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " Nov 02 18:12:29.962235 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.961823 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee859b4-abd7-4add-8511-1d4e2963e182-kube-api-access-xk4vb\" (UniqueName: \"kubernetes.io/projected/5ee859b4-abd7-4add-8511-1d4e2963e182-kube-api-access-xk4vb\") pod \"5ee859b4-abd7-4add-8511-1d4e2963e182\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " Nov 02 18:12:29.962235 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.961729 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-data" pod "5ee859b4-abd7-4add-8511-1d4e2963e182" (UID: "5ee859b4-abd7-4add-8511-1d4e2963e182"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:29.964220 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.964167 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee859b4-abd7-4add-8511-1d4e2963e182-istiod-ca-cert" pod "5ee859b4-abd7-4add-8511-1d4e2963e182" (UID: "5ee859b4-abd7-4add-8511-1d4e2963e182"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:29.964371 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.964260 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-workload-certs\") pod \"5ee859b4-abd7-4add-8511-1d4e2963e182\" (UID: \"5ee859b4-abd7-4add-8511-1d4e2963e182\") " Nov 02 18:12:29.964939 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.964910 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-envoy" pod "5ee859b4-abd7-4add-8511-1d4e2963e182" (UID: "5ee859b4-abd7-4add-8511-1d4e2963e182"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:29.967142 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.967119 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-workload-certs" pod "5ee859b4-abd7-4add-8511-1d4e2963e182" (UID: "5ee859b4-abd7-4add-8511-1d4e2963e182"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:29.967366 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.967339 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee859b4-abd7-4add-8511-1d4e2963e182-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:29.968067 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.968049 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:29.970486 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.968159 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:29.970486 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.968185 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:29.970486 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.968211 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:29.997895 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:29.997022 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-token" pod "5ee859b4-abd7-4add-8511-1d4e2963e182" (UID: "5ee859b4-abd7-4add-8511-1d4e2963e182"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:30.001945 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:30.001899 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee859b4-abd7-4add-8511-1d4e2963e182-kube-api-access-xk4vb" pod "5ee859b4-abd7-4add-8511-1d4e2963e182" (UID: "5ee859b4-abd7-4add-8511-1d4e2963e182"). InnerVolumeSpecName "kube-api-access-xk4vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:30.003324 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:30.003296 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-podinfo" pod "5ee859b4-abd7-4add-8511-1d4e2963e182" (UID: "5ee859b4-abd7-4add-8511-1d4e2963e182"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:12:30.070542 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:30.070504 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.070731 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:30.070549 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk4vb\" (UniqueName: \"kubernetes.io/projected/5ee859b4-abd7-4add-8511-1d4e2963e182-kube-api-access-xk4vb\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.070731 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:30.070565 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee859b4-abd7-4add-8511-1d4e2963e182-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.070731 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:30.070580 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee859b4-abd7-4add-8511-1d4e2963e182-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:30.495696 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:30.495653 5601 generic.go:356] "Generic (PLEG): container finished" podID="5ee859b4-abd7-4add-8511-1d4e2963e182" containerID="3a69163fac823f5a9926d1fabcac3ccaa3706a44d8c05cef3961ec2be3e0d4be" exitCode=0 Nov 02 18:12:30.495900 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:30.495698 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" event={"ID":"5ee859b4-abd7-4add-8511-1d4e2963e182","Type":"ContainerDied","Data":"3a69163fac823f5a9926d1fabcac3ccaa3706a44d8c05cef3961ec2be3e0d4be"} Nov 02 18:12:30.495900 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:30.495730 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" Nov 02 18:12:30.495900 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:30.495771 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp" event={"ID":"5ee859b4-abd7-4add-8511-1d4e2963e182","Type":"ContainerDied","Data":"5a9d5ef7b81b8e418fc9db6f4cd4bb7cdca2dae3338959256a5893f9582d6e5a"} Nov 02 18:12:30.495900 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:30.495805 5601 scope.go:122] "RemoveContainer" containerID="3a69163fac823f5a9926d1fabcac3ccaa3706a44d8c05cef3961ec2be3e0d4be" Nov 02 18:12:30.507927 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:30.507910 5601 scope.go:122] "RemoveContainer" containerID="3a69163fac823f5a9926d1fabcac3ccaa3706a44d8c05cef3961ec2be3e0d4be" Nov 02 18:12:30.508305 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:12:30.508263 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a69163fac823f5a9926d1fabcac3ccaa3706a44d8c05cef3961ec2be3e0d4be\": container with ID starting with 3a69163fac823f5a9926d1fabcac3ccaa3706a44d8c05cef3961ec2be3e0d4be not found: ID does not exist" containerID="3a69163fac823f5a9926d1fabcac3ccaa3706a44d8c05cef3961ec2be3e0d4be" Nov 02 18:12:30.508368 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:30.508313 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a69163fac823f5a9926d1fabcac3ccaa3706a44d8c05cef3961ec2be3e0d4be"} err="failed to get container status \"3a69163fac823f5a9926d1fabcac3ccaa3706a44d8c05cef3961ec2be3e0d4be\": rpc error: code = NotFound desc = could not find container \"3a69163fac823f5a9926d1fabcac3ccaa3706a44d8c05cef3961ec2be3e0d4be\": container with ID starting with 3a69163fac823f5a9926d1fabcac3ccaa3706a44d8c05cef3961ec2be3e0d4be not found: ID does not exist" Nov 02 18:12:30.524336 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:30.524314 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp"] Nov 02 18:12:30.528162 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:30.528143 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-j4vtp"] Nov 02 18:12:31.449015 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:31.448972 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee859b4-abd7-4add-8511-1d4e2963e182" path="/var/lib/kubelet/pods/5ee859b4-abd7-4add-8511-1d4e2963e182/volumes" Nov 02 18:12:32.744094 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:32.744064 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb"] Nov 02 18:12:32.744521 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:32.744287 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" podUID="b37ac056-d4aa-4c98-b1c0-aeb3436786b2" containerName="istio-proxy" containerID="cri-o://61aae5c336eb689d732e1af61c38aebc21d3f4ba0a6152393cdee38b301fc5c6" gracePeriod=30 Nov 02 18:12:34.504340 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:12:34.504305 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:12:38.137023 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.136993 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 18:12:38.291994 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.291962 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istiod-ca-cert\") pod \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " Nov 02 18:12:38.292172 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.292001 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-token\" (UniqueName: \"kubernetes.io/projected/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-token\") pod \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " Nov 02 18:12:38.292172 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.292057 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-kube-api-access-7zp6g\" (UniqueName: \"kubernetes.io/projected/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-kube-api-access-7zp6g\") pod \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " Nov 02 18:12:38.292172 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.292118 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-credential-socket\") pod \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " Nov 02 18:12:38.292172 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.292150 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-workload-certs\") pod \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " Nov 02 18:12:38.292408 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.292177 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-podinfo\") pod \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " Nov 02 18:12:38.292408 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.292252 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-data\") pod \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " Nov 02 18:12:38.292526 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.292423 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-envoy\") pod \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " Nov 02 18:12:38.292526 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.292455 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-workload-socket\") pod \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\" (UID: \"b37ac056-d4aa-4c98-b1c0-aeb3436786b2\") " Nov 02 18:12:38.292526 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.292453 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istiod-ca-cert" pod "b37ac056-d4aa-4c98-b1c0-aeb3436786b2" (UID: "b37ac056-d4aa-4c98-b1c0-aeb3436786b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:38.292752 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.292586 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-data" pod "b37ac056-d4aa-4c98-b1c0-aeb3436786b2" (UID: "b37ac056-d4aa-4c98-b1c0-aeb3436786b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:38.292752 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.292682 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-workload-socket" pod "b37ac056-d4aa-4c98-b1c0-aeb3436786b2" (UID: "b37ac056-d4aa-4c98-b1c0-aeb3436786b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:38.293293 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.292745 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-workload-certs" pod "b37ac056-d4aa-4c98-b1c0-aeb3436786b2" (UID: "b37ac056-d4aa-4c98-b1c0-aeb3436786b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:38.293293 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.293030 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-credential-socket" pod "b37ac056-d4aa-4c98-b1c0-aeb3436786b2" (UID: "b37ac056-d4aa-4c98-b1c0-aeb3436786b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:38.293405 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.293378 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:38.293544 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.293408 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:38.293544 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.293423 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:38.293544 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.293451 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:38.293544 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.293465 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:38.295102 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.295076 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-envoy" pod "b37ac056-d4aa-4c98-b1c0-aeb3436786b2" (UID: "b37ac056-d4aa-4c98-b1c0-aeb3436786b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:38.295492 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.295466 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-token" pod "b37ac056-d4aa-4c98-b1c0-aeb3436786b2" (UID: "b37ac056-d4aa-4c98-b1c0-aeb3436786b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:38.295905 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.295878 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-podinfo" pod "b37ac056-d4aa-4c98-b1c0-aeb3436786b2" (UID: "b37ac056-d4aa-4c98-b1c0-aeb3436786b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:12:38.296037 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.296018 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-kube-api-access-7zp6g" pod "b37ac056-d4aa-4c98-b1c0-aeb3436786b2" (UID: "b37ac056-d4aa-4c98-b1c0-aeb3436786b2"). InnerVolumeSpecName "kube-api-access-7zp6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:38.395597 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.395512 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:38.395597 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.395552 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:38.395597 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.395569 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:38.395597 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.395581 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zp6g\" (UniqueName: \"kubernetes.io/projected/b37ac056-d4aa-4c98-b1c0-aeb3436786b2-kube-api-access-7zp6g\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:38.563189 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.563149 5601 generic.go:356] "Generic (PLEG): container finished" podID="b37ac056-d4aa-4c98-b1c0-aeb3436786b2" containerID="61aae5c336eb689d732e1af61c38aebc21d3f4ba0a6152393cdee38b301fc5c6" exitCode=0 Nov 02 18:12:38.563372 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.563199 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" Nov 02 18:12:38.563372 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.563246 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" event={"ID":"b37ac056-d4aa-4c98-b1c0-aeb3436786b2","Type":"ContainerDied","Data":"61aae5c336eb689d732e1af61c38aebc21d3f4ba0a6152393cdee38b301fc5c6"} Nov 02 18:12:38.563372 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.563308 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb" event={"ID":"b37ac056-d4aa-4c98-b1c0-aeb3436786b2","Type":"ContainerDied","Data":"a649fb2afa7ea1195ab33261fdbd93b3a9b240ed1a7e18241b60eb9f716b4a8d"} Nov 02 18:12:38.563372 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.563338 5601 scope.go:122] "RemoveContainer" containerID="61aae5c336eb689d732e1af61c38aebc21d3f4ba0a6152393cdee38b301fc5c6" Nov 02 18:12:38.575612 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.575587 5601 scope.go:122] "RemoveContainer" containerID="61aae5c336eb689d732e1af61c38aebc21d3f4ba0a6152393cdee38b301fc5c6" Nov 02 18:12:38.575984 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:12:38.575951 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61aae5c336eb689d732e1af61c38aebc21d3f4ba0a6152393cdee38b301fc5c6\": container with ID starting with 61aae5c336eb689d732e1af61c38aebc21d3f4ba0a6152393cdee38b301fc5c6 not found: ID does not exist" containerID="61aae5c336eb689d732e1af61c38aebc21d3f4ba0a6152393cdee38b301fc5c6" Nov 02 18:12:38.576107 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.575993 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61aae5c336eb689d732e1af61c38aebc21d3f4ba0a6152393cdee38b301fc5c6"} err="failed to get container status \"61aae5c336eb689d732e1af61c38aebc21d3f4ba0a6152393cdee38b301fc5c6\": rpc error: code = NotFound desc = could not find container \"61aae5c336eb689d732e1af61c38aebc21d3f4ba0a6152393cdee38b301fc5c6\": container with ID starting with 61aae5c336eb689d732e1af61c38aebc21d3f4ba0a6152393cdee38b301fc5c6 not found: ID does not exist" Nov 02 18:12:38.597263 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.597233 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb"] Nov 02 18:12:38.602401 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:38.602377 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-bdsqb"] Nov 02 18:12:39.428345 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:12:39.428314 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37ac056-d4aa-4c98-b1c0-aeb3436786b2" path="/var/lib/kubelet/pods/b37ac056-d4aa-4c98-b1c0-aeb3436786b2/volumes" Nov 02 18:13:04.696764 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:13:04.696715 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:13:34.486864 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:13:34.486825 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:13:59.857502 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:13:59.857454 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl"] Nov 02 18:13:59.858755 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:13:59.858725 5601 cpu_manager.go:414] "containerMap: removing container" podUID="b37ac056-d4aa-4c98-b1c0-aeb3436786b2" containerName="istio-proxy" Nov 02 18:13:59.858755 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:13:59.858753 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37ac056-d4aa-4c98-b1c0-aeb3436786b2" containerName="istio-proxy" Nov 02 18:13:59.858953 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:13:59.858786 5601 cpu_manager.go:414] "containerMap: removing container" podUID="5ee859b4-abd7-4add-8511-1d4e2963e182" containerName="istio-proxy" Nov 02 18:13:59.858953 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:13:59.858794 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee859b4-abd7-4add-8511-1d4e2963e182" containerName="istio-proxy" Nov 02 18:13:59.859039 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:13:59.858979 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37ac056-d4aa-4c98-b1c0-aeb3436786b2" containerName="istio-proxy" Nov 02 18:13:59.859039 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:13:59.858997 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee859b4-abd7-4add-8511-1d4e2963e182" containerName="istio-proxy" Nov 02 18:13:59.871442 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:13:59.871405 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl"] Nov 02 18:13:59.871675 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:13:59.871539 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:13:59.902082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:13:59.902050 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-658b8c-openshift-default-dockercfg-l7ztd\"" Nov 02 18:13:59.989692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:13:59.985489 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv8mn\" (UniqueName: \"kubernetes.io/projected/84ed3fb1-d6b4-469c-b7a3-8af367d47208-kube-api-access-sv8mn\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:13:59.990161 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:13:59.990115 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-envoy\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.006660 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.003013 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-credential-socket\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.006660 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.003217 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-workload-certs\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.006660 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.003314 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istiod-ca-cert\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.006660 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.003351 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-token\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.006660 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.003420 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-podinfo\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.006660 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.003530 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-workload-socket\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.006660 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.003600 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-data\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.113205 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.113117 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istiod-ca-cert\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.113454 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.112064 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istiod-ca-cert\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.113659 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.113640 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-token\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.114383 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.114353 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-podinfo\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.115203 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.115183 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-workload-socket\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.115417 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.115398 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-data\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.115592 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.115576 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv8mn\" (UniqueName: \"kubernetes.io/projected/84ed3fb1-d6b4-469c-b7a3-8af367d47208-kube-api-access-sv8mn\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.124336 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.124310 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-envoy\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.125525 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.118363 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-data\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.125783 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.125747 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-credential-socket\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.126348 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.126329 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-workload-certs\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.126661 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.118103 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-workload-socket\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.128259 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.127961 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-workload-certs\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.128259 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.128213 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-credential-socket\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.128259 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.128238 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-podinfo\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.150240 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.139303 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-token\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.153639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.151735 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv8mn\" (UniqueName: \"kubernetes.io/projected/84ed3fb1-d6b4-469c-b7a3-8af367d47208-kube-api-access-sv8mn\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.163641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.160018 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-envoy\") pod \"e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.192638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.189528 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:00.679860 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:14:00.679829 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ed3fb1_d6b4_469c_b7a3_8af367d47208.slice/crio-2f41205b2beee590c57d7e6aaa8b972f5113e6822dcef6625ecb7e9a0b3e3681 WatchSource:0}: Error finding container 2f41205b2beee590c57d7e6aaa8b972f5113e6822dcef6625ecb7e9a0b3e3681: Status 404 returned error can't find the container with id 2f41205b2beee590c57d7e6aaa8b972f5113e6822dcef6625ecb7e9a0b3e3681 Nov 02 18:14:00.682323 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.682292 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl"] Nov 02 18:14:00.682912 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.682880 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:14:00.682999 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.682948 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:14:00.682999 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.682977 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:14:00.800637 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.800245 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" event={"ID":"84ed3fb1-d6b4-469c-b7a3-8af367d47208","Type":"ContainerStarted","Data":"75ce8e3cf73639828799ca59a1c1271b21f787c6c1d6803312638ee5ef5db6ff"} Nov 02 18:14:00.800637 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.800307 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" event={"ID":"84ed3fb1-d6b4-469c-b7a3-8af367d47208","Type":"ContainerStarted","Data":"2f41205b2beee590c57d7e6aaa8b972f5113e6822dcef6625ecb7e9a0b3e3681"} Nov 02 18:14:00.832935 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:00.832825 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" podStartSLOduration=1.832801154 podStartE2EDuration="1.832801154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:00.822214665 +0000 UTC m=+2405.513268678" watchObservedRunningTime="2026-11-02 18:14:00.832801154 +0000 UTC m=+2405.523855169" Nov 02 18:14:01.190489 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:01.190450 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:01.192763 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:01.192731 5601 patch_prober.go:29] interesting pod/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 02 18:14:01.192910 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:01.192792 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" podUID="84ed3fb1-d6b4-469c-b7a3-8af367d47208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 02 18:14:02.195131 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:02.195096 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:02.817358 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:02.817323 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:02.818398 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:02.818370 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:14:04.751098 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:14:04.751057 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:14:34.487899 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:14:34.487853 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:14:58.622992 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:58.622909 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf"] Nov 02 18:14:58.623501 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:58.623175 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" podUID="8d603436-a2a3-41f2-8554-c654ac4a9a45" containerName="istio-proxy" containerID="cri-o://b9165dc79af6077d94eafb7e850e87f649f1903f4719bacdeef18de7b10eef95" gracePeriod=30 Nov 02 18:14:58.838656 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:58.838570 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2"] Nov 02 18:14:59.419649 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:14:59.416971 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl"] Nov 02 18:15:00.201490 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.201440 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf"] Nov 02 18:15:00.209564 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.207544 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" Nov 02 18:15:00.234532 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.234498 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf"] Nov 02 18:15:00.241797 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.241757 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z"] Nov 02 18:15:00.250872 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.250843 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z" Nov 02 18:15:00.261501 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.261423 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18f2b645-9cdf-4b3e-8b54-8e31706d33a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf\" (UID: \"18f2b645-9cdf-4b3e-8b54-8e31706d33a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" Nov 02 18:15:00.261804 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.261780 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmk7d\" (UniqueName: \"kubernetes.io/projected/e6aed869-903a-4e44-aaed-66d8940f4f90-kube-api-access-nmk7d\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z\" (UID: \"e6aed869-903a-4e44-aaed-66d8940f4f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z" Nov 02 18:15:00.262180 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.262157 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\"" Nov 02 18:15:00.262250 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.262155 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzs7t\" (UniqueName: \"kubernetes.io/projected/18f2b645-9cdf-4b3e-8b54-8e31706d33a3-kube-api-access-zzs7t\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf\" (UID: \"18f2b645-9cdf-4b3e-8b54-8e31706d33a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" Nov 02 18:15:00.262390 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.262371 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6aed869-903a-4e44-aaed-66d8940f4f90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z\" (UID: \"e6aed869-903a-4e44-aaed-66d8940f4f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z" Nov 02 18:15:00.264477 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.264442 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj"] Nov 02 18:15:00.274720 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.274162 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" Nov 02 18:15:00.333588 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.333552 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s"] Nov 02 18:15:00.340989 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.340850 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt"] Nov 02 18:15:00.343602 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.340956 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s" Nov 02 18:15:00.348611 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.348559 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j"] Nov 02 18:15:00.350879 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.350854 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt" Nov 02 18:15:00.356549 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.356501 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j" Nov 02 18:15:00.360561 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.360527 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z"] Nov 02 18:15:00.367649 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.366533 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588"] Nov 02 18:15:00.385869 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.385840 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588" Nov 02 18:15:00.408171 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.408128 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6aed869-903a-4e44-aaed-66d8940f4f90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z\" (UID: \"e6aed869-903a-4e44-aaed-66d8940f4f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z" Nov 02 18:15:00.408566 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.408543 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8pxg\" (UniqueName: \"kubernetes.io/projected/aa194666-bb2c-4610-a051-7fd9caef77d3-kube-api-access-g8pxg\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s\" (UID: \"aa194666-bb2c-4610-a051-7fd9caef77d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s" Nov 02 18:15:00.408771 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.408753 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q795d\" (UniqueName: \"kubernetes.io/projected/154aa781-e85c-4aad-8fc1-e022e9704801-kube-api-access-q795d\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt\" (UID: \"154aa781-e85c-4aad-8fc1-e022e9704801\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt" Nov 02 18:15:00.408952 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.408925 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18f2b645-9cdf-4b3e-8b54-8e31706d33a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf\" (UID: \"18f2b645-9cdf-4b3e-8b54-8e31706d33a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" Nov 02 18:15:00.409098 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.409083 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmk7d\" (UniqueName: \"kubernetes.io/projected/e6aed869-903a-4e44-aaed-66d8940f4f90-kube-api-access-nmk7d\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z\" (UID: \"e6aed869-903a-4e44-aaed-66d8940f4f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z" Nov 02 18:15:00.409297 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.409281 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnkxw\" (UniqueName: \"kubernetes.io/projected/e2f91fe1-6e31-4c8c-a24e-2d473a179960-kube-api-access-mnkxw\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj\" (UID: \"e2f91fe1-6e31-4c8c-a24e-2d473a179960\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" Nov 02 18:15:00.409459 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.409445 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/154aa781-e85c-4aad-8fc1-e022e9704801-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt\" (UID: \"154aa781-e85c-4aad-8fc1-e022e9704801\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt" Nov 02 18:15:00.409586 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.409573 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f91fe1-6e31-4c8c-a24e-2d473a179960-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj\" (UID: \"e2f91fe1-6e31-4c8c-a24e-2d473a179960\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" Nov 02 18:15:00.409830 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.409816 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa194666-bb2c-4610-a051-7fd9caef77d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s\" (UID: \"aa194666-bb2c-4610-a051-7fd9caef77d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s" Nov 02 18:15:00.409946 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.409933 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzs7t\" (UniqueName: \"kubernetes.io/projected/18f2b645-9cdf-4b3e-8b54-8e31706d33a3-kube-api-access-zzs7t\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf\" (UID: \"18f2b645-9cdf-4b3e-8b54-8e31706d33a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" Nov 02 18:15:00.427759 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.414395 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18f2b645-9cdf-4b3e-8b54-8e31706d33a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf\" (UID: \"18f2b645-9cdf-4b3e-8b54-8e31706d33a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" Nov 02 18:15:00.427759 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.415080 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6aed869-903a-4e44-aaed-66d8940f4f90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z\" (UID: \"e6aed869-903a-4e44-aaed-66d8940f4f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z" Nov 02 18:15:00.460728 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.460632 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj"] Nov 02 18:15:00.469734 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.469690 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s"] Nov 02 18:15:00.478563 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.478534 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmk7d\" (UniqueName: \"kubernetes.io/projected/e6aed869-903a-4e44-aaed-66d8940f4f90-kube-api-access-nmk7d\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z\" (UID: \"e6aed869-903a-4e44-aaed-66d8940f4f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z" Nov 02 18:15:00.482335 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.482277 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzs7t\" (UniqueName: \"kubernetes.io/projected/18f2b645-9cdf-4b3e-8b54-8e31706d33a3-kube-api-access-zzs7t\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf\" (UID: \"18f2b645-9cdf-4b3e-8b54-8e31706d33a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" Nov 02 18:15:00.485134 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.485056 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt"] Nov 02 18:15:00.520077 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.507848 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb"] Nov 02 18:15:00.520077 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.513552 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnkxw\" (UniqueName: \"kubernetes.io/projected/e2f91fe1-6e31-4c8c-a24e-2d473a179960-kube-api-access-mnkxw\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj\" (UID: \"e2f91fe1-6e31-4c8c-a24e-2d473a179960\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" Nov 02 18:15:00.520077 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.513660 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/154aa781-e85c-4aad-8fc1-e022e9704801-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt\" (UID: \"154aa781-e85c-4aad-8fc1-e022e9704801\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt" Nov 02 18:15:00.520077 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.513719 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f91fe1-6e31-4c8c-a24e-2d473a179960-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj\" (UID: \"e2f91fe1-6e31-4c8c-a24e-2d473a179960\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" Nov 02 18:15:00.520077 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.513862 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa194666-bb2c-4610-a051-7fd9caef77d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s\" (UID: \"aa194666-bb2c-4610-a051-7fd9caef77d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s" Nov 02 18:15:00.520077 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.513909 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6612ae11-596f-4056-b6bc-42ed18333072-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j\" (UID: \"6612ae11-596f-4056-b6bc-42ed18333072\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j" Nov 02 18:15:00.520077 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.513994 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8pxg\" (UniqueName: \"kubernetes.io/projected/aa194666-bb2c-4610-a051-7fd9caef77d3-kube-api-access-g8pxg\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s\" (UID: \"aa194666-bb2c-4610-a051-7fd9caef77d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s" Nov 02 18:15:00.520077 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.514080 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q795d\" (UniqueName: \"kubernetes.io/projected/154aa781-e85c-4aad-8fc1-e022e9704801-kube-api-access-q795d\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt\" (UID: \"154aa781-e85c-4aad-8fc1-e022e9704801\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt" Nov 02 18:15:00.520077 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.514182 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78898\" (UniqueName: \"kubernetes.io/projected/6612ae11-596f-4056-b6bc-42ed18333072-kube-api-access-78898\") pod \"maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j\" (UID: \"6612ae11-596f-4056-b6bc-42ed18333072\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j" Nov 02 18:15:00.525546 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.523723 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/154aa781-e85c-4aad-8fc1-e022e9704801-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt\" (UID: \"154aa781-e85c-4aad-8fc1-e022e9704801\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt" Nov 02 18:15:00.525546 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.524404 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f91fe1-6e31-4c8c-a24e-2d473a179960-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj\" (UID: \"e2f91fe1-6e31-4c8c-a24e-2d473a179960\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" Nov 02 18:15:00.525546 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.524823 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa194666-bb2c-4610-a051-7fd9caef77d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s\" (UID: \"aa194666-bb2c-4610-a051-7fd9caef77d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s" Nov 02 18:15:00.525906 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.525885 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r"] Nov 02 18:15:00.526707 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.526202 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb" Nov 02 18:15:00.536264 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.536227 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8pxg\" (UniqueName: \"kubernetes.io/projected/aa194666-bb2c-4610-a051-7fd9caef77d3-kube-api-access-g8pxg\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s\" (UID: \"aa194666-bb2c-4610-a051-7fd9caef77d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s" Nov 02 18:15:00.537710 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.536773 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnkxw\" (UniqueName: \"kubernetes.io/projected/e2f91fe1-6e31-4c8c-a24e-2d473a179960-kube-api-access-mnkxw\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj\" (UID: \"e2f91fe1-6e31-4c8c-a24e-2d473a179960\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" Nov 02 18:15:00.537710 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.537676 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj"] Nov 02 18:15:00.537912 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.537746 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r" Nov 02 18:15:00.539682 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.539605 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q795d\" (UniqueName: \"kubernetes.io/projected/154aa781-e85c-4aad-8fc1-e022e9704801-kube-api-access-q795d\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt\" (UID: \"154aa781-e85c-4aad-8fc1-e022e9704801\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt" Nov 02 18:15:00.542993 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.542969 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7"] Nov 02 18:15:00.543128 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.543102 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" Nov 02 18:15:00.549351 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.549281 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588"] Nov 02 18:15:00.549509 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.549376 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" Nov 02 18:15:00.550948 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.550917 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r"] Nov 02 18:15:00.554122 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.554096 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" Nov 02 18:15:00.562848 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.562818 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb"] Nov 02 18:15:00.563068 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.563046 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg"] Nov 02 18:15:00.568090 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.568066 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps"] Nov 02 18:15:00.568256 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.568189 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg" Nov 02 18:15:00.572765 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.572742 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx"] Nov 02 18:15:00.572897 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.572841 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" Nov 02 18:15:00.577454 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.577433 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7"] Nov 02 18:15:00.577604 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.577582 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" Nov 02 18:15:00.580047 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.580023 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps"] Nov 02 18:15:00.586971 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.586939 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z" Nov 02 18:15:00.589611 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.589528 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j"] Nov 02 18:15:00.603748 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.603269 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj"] Nov 02 18:15:00.612840 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.612513 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" Nov 02 18:15:00.612840 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.612690 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg"] Nov 02 18:15:00.617256 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.615370 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78898\" (UniqueName: \"kubernetes.io/projected/6612ae11-596f-4056-b6bc-42ed18333072-kube-api-access-78898\") pod \"maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j\" (UID: \"6612ae11-596f-4056-b6bc-42ed18333072\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j" Nov 02 18:15:00.617256 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.615766 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjgsw\" (UniqueName: \"kubernetes.io/projected/23af2a44-3074-4dce-a5c6-9c18b4c87d35-kube-api-access-wjgsw\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588\" (UID: \"23af2a44-3074-4dce-a5c6-9c18b4c87d35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588" Nov 02 18:15:00.617256 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.616020 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6612ae11-596f-4056-b6bc-42ed18333072-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j\" (UID: \"6612ae11-596f-4056-b6bc-42ed18333072\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j" Nov 02 18:15:00.617256 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.616054 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23af2a44-3074-4dce-a5c6-9c18b4c87d35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588\" (UID: \"23af2a44-3074-4dce-a5c6-9c18b4c87d35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588" Nov 02 18:15:00.617256 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.616900 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6612ae11-596f-4056-b6bc-42ed18333072-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j\" (UID: \"6612ae11-596f-4056-b6bc-42ed18333072\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j" Nov 02 18:15:00.620096 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.620046 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx"] Nov 02 18:15:00.629095 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.629038 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78898\" (UniqueName: \"kubernetes.io/projected/6612ae11-596f-4056-b6bc-42ed18333072-kube-api-access-78898\") pod \"maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j\" (UID: \"6612ae11-596f-4056-b6bc-42ed18333072\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j" Nov 02 18:15:00.698809 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.698552 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s" Nov 02 18:15:00.705970 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.705935 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt" Nov 02 18:15:00.719929 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.719894 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b2mv\" (UniqueName: \"kubernetes.io/projected/3e296182-a14a-4ee7-9cd2-d014826b6cd9-kube-api-access-5b2mv\") pod \"maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7\" (UID: \"3e296182-a14a-4ee7-9cd2-d014826b6cd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" Nov 02 18:15:00.720077 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.719982 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2ceb92-645d-4255-89a8-c0211bc08025-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj\" (UID: \"bd2ceb92-645d-4255-89a8-c0211bc08025\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" Nov 02 18:15:00.720137 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.720105 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e45226-6362-4b36-b890-11b6804c8d65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx\" (UID: \"c4e45226-6362-4b36-b890-11b6804c8d65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" Nov 02 18:15:00.720226 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.720161 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611e836-f87d-448f-8f33-5f58724e18cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg\" (UID: \"3611e836-f87d-448f-8f33-5f58724e18cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg" Nov 02 18:15:00.720226 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.720209 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23af2a44-3074-4dce-a5c6-9c18b4c87d35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588\" (UID: \"23af2a44-3074-4dce-a5c6-9c18b4c87d35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588" Nov 02 18:15:00.720336 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.720271 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e296182-a14a-4ee7-9cd2-d014826b6cd9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7\" (UID: \"3e296182-a14a-4ee7-9cd2-d014826b6cd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" Nov 02 18:15:00.734856 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.720397 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p87rq\" (UniqueName: \"kubernetes.io/projected/c4e45226-6362-4b36-b890-11b6804c8d65-kube-api-access-p87rq\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx\" (UID: \"c4e45226-6362-4b36-b890-11b6804c8d65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" Nov 02 18:15:00.734856 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.720506 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn94r\" (UniqueName: \"kubernetes.io/projected/b16fa692-4b98-42ad-b4db-b38d411c093a-kube-api-access-xn94r\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r\" (UID: \"b16fa692-4b98-42ad-b4db-b38d411c093a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r" Nov 02 18:15:00.734856 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.720558 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvj9b\" (UniqueName: \"kubernetes.io/projected/871bed77-cbe7-4001-a4f8-a6cedca05cef-kube-api-access-qvj9b\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps\" (UID: \"871bed77-cbe7-4001-a4f8-a6cedca05cef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" Nov 02 18:15:00.734856 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.721517 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t52l7\" (UniqueName: \"kubernetes.io/projected/62588dd0-b7ee-440e-ab7e-419d7429b2d9-kube-api-access-t52l7\") pod \"maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb\" (UID: \"62588dd0-b7ee-440e-ab7e-419d7429b2d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb" Nov 02 18:15:00.734856 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.721575 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62588dd0-b7ee-440e-ab7e-419d7429b2d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb\" (UID: \"62588dd0-b7ee-440e-ab7e-419d7429b2d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb" Nov 02 18:15:00.734856 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.721640 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/871bed77-cbe7-4001-a4f8-a6cedca05cef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps\" (UID: \"871bed77-cbe7-4001-a4f8-a6cedca05cef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" Nov 02 18:15:00.734856 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.721736 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrqjq\" (UniqueName: \"kubernetes.io/projected/bd2ceb92-645d-4255-89a8-c0211bc08025-kube-api-access-rrqjq\") pod \"maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj\" (UID: \"bd2ceb92-645d-4255-89a8-c0211bc08025\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" Nov 02 18:15:00.734856 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.721783 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16fa692-4b98-42ad-b4db-b38d411c093a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r\" (UID: \"b16fa692-4b98-42ad-b4db-b38d411c093a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r" Nov 02 18:15:00.734856 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.721908 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjgsw\" (UniqueName: \"kubernetes.io/projected/23af2a44-3074-4dce-a5c6-9c18b4c87d35-kube-api-access-wjgsw\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588\" (UID: \"23af2a44-3074-4dce-a5c6-9c18b4c87d35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588" Nov 02 18:15:00.734856 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.721948 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz85d\" (UniqueName: \"kubernetes.io/projected/3611e836-f87d-448f-8f33-5f58724e18cf-kube-api-access-pz85d\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg\" (UID: \"3611e836-f87d-448f-8f33-5f58724e18cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg" Nov 02 18:15:00.734856 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.722017 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23af2a44-3074-4dce-a5c6-9c18b4c87d35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588\" (UID: \"23af2a44-3074-4dce-a5c6-9c18b4c87d35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588" Nov 02 18:15:00.745743 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.745710 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjgsw\" (UniqueName: \"kubernetes.io/projected/23af2a44-3074-4dce-a5c6-9c18b4c87d35-kube-api-access-wjgsw\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588\" (UID: \"23af2a44-3074-4dce-a5c6-9c18b4c87d35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588" Nov 02 18:15:00.781392 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.781355 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j" Nov 02 18:15:00.837866 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.837367 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn94r\" (UniqueName: \"kubernetes.io/projected/b16fa692-4b98-42ad-b4db-b38d411c093a-kube-api-access-xn94r\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r\" (UID: \"b16fa692-4b98-42ad-b4db-b38d411c093a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r" Nov 02 18:15:00.837866 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.837441 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvj9b\" (UniqueName: \"kubernetes.io/projected/871bed77-cbe7-4001-a4f8-a6cedca05cef-kube-api-access-qvj9b\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps\" (UID: \"871bed77-cbe7-4001-a4f8-a6cedca05cef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" Nov 02 18:15:00.837866 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.837477 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t52l7\" (UniqueName: \"kubernetes.io/projected/62588dd0-b7ee-440e-ab7e-419d7429b2d9-kube-api-access-t52l7\") pod \"maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb\" (UID: \"62588dd0-b7ee-440e-ab7e-419d7429b2d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb" Nov 02 18:15:00.837866 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.837518 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62588dd0-b7ee-440e-ab7e-419d7429b2d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb\" (UID: \"62588dd0-b7ee-440e-ab7e-419d7429b2d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb" Nov 02 18:15:00.837866 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.837562 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/871bed77-cbe7-4001-a4f8-a6cedca05cef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps\" (UID: \"871bed77-cbe7-4001-a4f8-a6cedca05cef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" Nov 02 18:15:00.837866 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.837609 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrqjq\" (UniqueName: \"kubernetes.io/projected/bd2ceb92-645d-4255-89a8-c0211bc08025-kube-api-access-rrqjq\") pod \"maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj\" (UID: \"bd2ceb92-645d-4255-89a8-c0211bc08025\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" Nov 02 18:15:00.837866 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.837669 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16fa692-4b98-42ad-b4db-b38d411c093a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r\" (UID: \"b16fa692-4b98-42ad-b4db-b38d411c093a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r" Nov 02 18:15:00.837866 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.837750 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz85d\" (UniqueName: \"kubernetes.io/projected/3611e836-f87d-448f-8f33-5f58724e18cf-kube-api-access-pz85d\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg\" (UID: \"3611e836-f87d-448f-8f33-5f58724e18cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg" Nov 02 18:15:00.837866 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.837825 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b2mv\" (UniqueName: \"kubernetes.io/projected/3e296182-a14a-4ee7-9cd2-d014826b6cd9-kube-api-access-5b2mv\") pod \"maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7\" (UID: \"3e296182-a14a-4ee7-9cd2-d014826b6cd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" Nov 02 18:15:00.838454 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.837895 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2ceb92-645d-4255-89a8-c0211bc08025-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj\" (UID: \"bd2ceb92-645d-4255-89a8-c0211bc08025\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" Nov 02 18:15:00.838454 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.838023 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e45226-6362-4b36-b890-11b6804c8d65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx\" (UID: \"c4e45226-6362-4b36-b890-11b6804c8d65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" Nov 02 18:15:00.838454 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.838080 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611e836-f87d-448f-8f33-5f58724e18cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg\" (UID: \"3611e836-f87d-448f-8f33-5f58724e18cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg" Nov 02 18:15:00.838454 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.838134 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e296182-a14a-4ee7-9cd2-d014826b6cd9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7\" (UID: \"3e296182-a14a-4ee7-9cd2-d014826b6cd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" Nov 02 18:15:00.838454 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.838258 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p87rq\" (UniqueName: \"kubernetes.io/projected/c4e45226-6362-4b36-b890-11b6804c8d65-kube-api-access-p87rq\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx\" (UID: \"c4e45226-6362-4b36-b890-11b6804c8d65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" Nov 02 18:15:00.840028 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.839998 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62588dd0-b7ee-440e-ab7e-419d7429b2d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb\" (UID: \"62588dd0-b7ee-440e-ab7e-419d7429b2d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb" Nov 02 18:15:00.840822 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.840794 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/871bed77-cbe7-4001-a4f8-a6cedca05cef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps\" (UID: \"871bed77-cbe7-4001-a4f8-a6cedca05cef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" Nov 02 18:15:00.841019 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.840989 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588" Nov 02 18:15:00.846815 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.844348 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16fa692-4b98-42ad-b4db-b38d411c093a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r\" (UID: \"b16fa692-4b98-42ad-b4db-b38d411c093a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r" Nov 02 18:15:00.846815 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.845199 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e45226-6362-4b36-b890-11b6804c8d65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx\" (UID: \"c4e45226-6362-4b36-b890-11b6804c8d65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" Nov 02 18:15:00.846815 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.845837 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2ceb92-645d-4255-89a8-c0211bc08025-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj\" (UID: \"bd2ceb92-645d-4255-89a8-c0211bc08025\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" Nov 02 18:15:00.846815 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.846452 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611e836-f87d-448f-8f33-5f58724e18cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg\" (UID: \"3611e836-f87d-448f-8f33-5f58724e18cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg" Nov 02 18:15:00.848012 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.847994 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e296182-a14a-4ee7-9cd2-d014826b6cd9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7\" (UID: \"3e296182-a14a-4ee7-9cd2-d014826b6cd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" Nov 02 18:15:00.871996 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.864698 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz85d\" (UniqueName: \"kubernetes.io/projected/3611e836-f87d-448f-8f33-5f58724e18cf-kube-api-access-pz85d\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg\" (UID: \"3611e836-f87d-448f-8f33-5f58724e18cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg" Nov 02 18:15:00.871996 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.865394 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrqjq\" (UniqueName: \"kubernetes.io/projected/bd2ceb92-645d-4255-89a8-c0211bc08025-kube-api-access-rrqjq\") pod \"maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj\" (UID: \"bd2ceb92-645d-4255-89a8-c0211bc08025\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" Nov 02 18:15:00.871996 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.865442 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf"] Nov 02 18:15:00.871996 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.866086 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvj9b\" (UniqueName: \"kubernetes.io/projected/871bed77-cbe7-4001-a4f8-a6cedca05cef-kube-api-access-qvj9b\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps\" (UID: \"871bed77-cbe7-4001-a4f8-a6cedca05cef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" Nov 02 18:15:00.871996 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.867092 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p87rq\" (UniqueName: \"kubernetes.io/projected/c4e45226-6362-4b36-b890-11b6804c8d65-kube-api-access-p87rq\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx\" (UID: \"c4e45226-6362-4b36-b890-11b6804c8d65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" Nov 02 18:15:00.871996 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.867907 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn94r\" (UniqueName: \"kubernetes.io/projected/b16fa692-4b98-42ad-b4db-b38d411c093a-kube-api-access-xn94r\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r\" (UID: \"b16fa692-4b98-42ad-b4db-b38d411c093a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r" Nov 02 18:15:00.871996 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.868601 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b2mv\" (UniqueName: \"kubernetes.io/projected/3e296182-a14a-4ee7-9cd2-d014826b6cd9-kube-api-access-5b2mv\") pod \"maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7\" (UID: \"3e296182-a14a-4ee7-9cd2-d014826b6cd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" Nov 02 18:15:00.871996 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.871363 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t52l7\" (UniqueName: \"kubernetes.io/projected/62588dd0-b7ee-440e-ab7e-419d7429b2d9-kube-api-access-t52l7\") pod \"maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb\" (UID: \"62588dd0-b7ee-440e-ab7e-419d7429b2d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb" Nov 02 18:15:00.920611 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.920577 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb" Nov 02 18:15:00.937250 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:00.937218 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r" Nov 02 18:15:01.022382 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:01.021989 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" Nov 02 18:15:01.048766 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:01.048489 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" Nov 02 18:15:01.087975 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:01.087938 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" Nov 02 18:15:01.090599 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:01.090568 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg" Nov 02 18:15:01.133996 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:01.133722 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" Nov 02 18:15:01.623801 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:01.623193 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z"] Nov 02 18:15:01.661287 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:01.638649 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj"] Nov 02 18:15:01.800628 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:01.800571 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z" event={"ID":"e6aed869-903a-4e44-aaed-66d8940f4f90","Type":"ContainerStarted","Data":"aa56341819cc5f9c6845f7b32fde80c6bc7cda1f69a6605c6c517f11f8c829a9"} Nov 02 18:15:01.808023 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:01.807927 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" event={"ID":"e2f91fe1-6e31-4c8c-a24e-2d473a179960","Type":"ContainerStarted","Data":"6f627a35a5e5cad3610b7e90278495c59c79e6a278090c690e07f361a44a36ca"} Nov 02 18:15:01.811734 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:01.811328 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" event={"ID":"18f2b645-9cdf-4b3e-8b54-8e31706d33a3","Type":"ContainerStarted","Data":"382f729da268d7107a5af93ae8380488924896aa24ef37069a42b16224b6fa09"} Nov 02 18:15:01.811734 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:01.811411 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" event={"ID":"18f2b645-9cdf-4b3e-8b54-8e31706d33a3","Type":"ContainerStarted","Data":"6fe6fe29490542fd05fa822117c5e89807b0bddc312340af46de7717833ade2c"} Nov 02 18:15:01.868666 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:01.867711 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" podStartSLOduration=1.8676883819999999 podStartE2EDuration="1.867688382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:01.861970208 +0000 UTC m=+2466.553024232" watchObservedRunningTime="2026-11-02 18:15:01.867688382 +0000 UTC m=+2466.558742395" Nov 02 18:15:01.984395 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:01.984351 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt"] Nov 02 18:15:01.986220 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:15:01.986137 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154aa781_e85c_4aad_8fc1_e022e9704801.slice/crio-6266d96251ad1f9a773294f581a8eb5c38b75df4dc02e9c098232fee982d43be WatchSource:0}: Error finding container 6266d96251ad1f9a773294f581a8eb5c38b75df4dc02e9c098232fee982d43be: Status 404 returned error can't find the container with id 6266d96251ad1f9a773294f581a8eb5c38b75df4dc02e9c098232fee982d43be Nov 02 18:15:01.991171 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:01.991145 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r"] Nov 02 18:15:01.996064 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:15:01.996039 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa194666_bb2c_4610_a051_7fd9caef77d3.slice/crio-71846bfab522d5f51d3a14b04387e76613f410230c6bed3b9d2ca576f99ad481 WatchSource:0}: Error finding container 71846bfab522d5f51d3a14b04387e76613f410230c6bed3b9d2ca576f99ad481: Status 404 returned error can't find the container with id 71846bfab522d5f51d3a14b04387e76613f410230c6bed3b9d2ca576f99ad481 Nov 02 18:15:01.999649 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:01.999609 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s"] Nov 02 18:15:02.008848 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:15:02.008790 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6612ae11_596f_4056_b6bc_42ed18333072.slice/crio-c93399e5267cf3e3153b482aa16b77740f472b17beb07049ff9accc567245355 WatchSource:0}: Error finding container c93399e5267cf3e3153b482aa16b77740f472b17beb07049ff9accc567245355: Status 404 returned error can't find the container with id c93399e5267cf3e3153b482aa16b77740f472b17beb07049ff9accc567245355 Nov 02 18:15:02.009507 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:02.009487 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j"] Nov 02 18:15:02.096985 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:02.096951 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588"] Nov 02 18:15:02.097320 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:15:02.097269 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23af2a44_3074_4dce_a5c6_9c18b4c87d35.slice/crio-d02b2000b0934f024242254fc1afd706b4f20ae7b89c2b1bc021f10a4781708a WatchSource:0}: Error finding container d02b2000b0934f024242254fc1afd706b4f20ae7b89c2b1bc021f10a4781708a: Status 404 returned error can't find the container with id d02b2000b0934f024242254fc1afd706b4f20ae7b89c2b1bc021f10a4781708a Nov 02 18:15:02.109492 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:02.109453 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb"] Nov 02 18:15:02.113673 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:15:02.113642 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62588dd0_b7ee_440e_ab7e_419d7429b2d9.slice/crio-50ff4ad8fd522c0e968a0339ca77a968336c04d0ff3c7dccddcab8039b1d3fac WatchSource:0}: Error finding container 50ff4ad8fd522c0e968a0339ca77a968336c04d0ff3c7dccddcab8039b1d3fac: Status 404 returned error can't find the container with id 50ff4ad8fd522c0e968a0339ca77a968336c04d0ff3c7dccddcab8039b1d3fac Nov 02 18:15:02.192660 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:02.184848 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg"] Nov 02 18:15:02.326079 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:02.321913 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7"] Nov 02 18:15:02.398836 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:02.347165 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj"] Nov 02 18:15:02.482367 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:02.465456 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps"] Nov 02 18:15:02.482367 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:15:02.469983 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3611e836_f87d_448f_8f33_5f58724e18cf.slice/crio-7fc6ae65a3d51b510a50eec6d4eaa3f30e9bce1352afb23233d421a5512d3377 WatchSource:0}: Error finding container 7fc6ae65a3d51b510a50eec6d4eaa3f30e9bce1352afb23233d421a5512d3377: Status 404 returned error can't find the container with id 7fc6ae65a3d51b510a50eec6d4eaa3f30e9bce1352afb23233d421a5512d3377 Nov 02 18:15:02.515530 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:02.502435 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx"] Nov 02 18:15:02.980259 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:02.980113 5601 generic.go:356] "Generic (PLEG): container finished" podID="18f2b645-9cdf-4b3e-8b54-8e31706d33a3" containerID="382f729da268d7107a5af93ae8380488924896aa24ef37069a42b16224b6fa09" exitCode=60 Nov 02 18:15:02.980891 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:02.980862 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" event={"ID":"18f2b645-9cdf-4b3e-8b54-8e31706d33a3","Type":"ContainerDied","Data":"382f729da268d7107a5af93ae8380488924896aa24ef37069a42b16224b6fa09"} Nov 02 18:15:02.981367 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:02.981346 5601 scope.go:122] "RemoveContainer" containerID="382f729da268d7107a5af93ae8380488924896aa24ef37069a42b16224b6fa09" Nov 02 18:15:03.012439 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.012367 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" event={"ID":"c4e45226-6362-4b36-b890-11b6804c8d65","Type":"ContainerStarted","Data":"638b66edb805930e04d92bd2929abea330ff9d976a6e9e92c6ae49aeb8a155b1"} Nov 02 18:15:03.014797 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.014763 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r" event={"ID":"b16fa692-4b98-42ad-b4db-b38d411c093a","Type":"ContainerStarted","Data":"fa26bd902d87483c89b45e9b7c0092b0831f393c2bea0a757301838dc0bc895a"} Nov 02 18:15:03.025181 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.024683 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" event={"ID":"3e296182-a14a-4ee7-9cd2-d014826b6cd9","Type":"ContainerStarted","Data":"0e85d0d33ac5be12bde3360d8fec44c4cac6ac6a64f88c765d5d44a19e4af4c9"} Nov 02 18:15:03.070665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.069323 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb" event={"ID":"62588dd0-b7ee-440e-ab7e-419d7429b2d9","Type":"ContainerStarted","Data":"50ff4ad8fd522c0e968a0339ca77a968336c04d0ff3c7dccddcab8039b1d3fac"} Nov 02 18:15:03.078890 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.078350 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r" podStartSLOduration=3.078329614 podStartE2EDuration="3.078329614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.0440055 +0000 UTC m=+2467.735059518" watchObservedRunningTime="2026-11-02 18:15:03.078329614 +0000 UTC m=+2467.769383628" Nov 02 18:15:03.085679 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.085052 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" event={"ID":"871bed77-cbe7-4001-a4f8-a6cedca05cef","Type":"ContainerStarted","Data":"31935b9fc170ab9708e683a341de5607561919f68ff92e8128c24b0c41eea579"} Nov 02 18:15:03.097966 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.097910 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb" podStartSLOduration=3.097877557 podStartE2EDuration="3.097877557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.096138941 +0000 UTC m=+2467.787192955" watchObservedRunningTime="2026-11-02 18:15:03.097877557 +0000 UTC m=+2467.788931571" Nov 02 18:15:03.099400 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.099168 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588" event={"ID":"23af2a44-3074-4dce-a5c6-9c18b4c87d35","Type":"ContainerStarted","Data":"d02b2000b0934f024242254fc1afd706b4f20ae7b89c2b1bc021f10a4781708a"} Nov 02 18:15:03.104102 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.103206 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt" event={"ID":"154aa781-e85c-4aad-8fc1-e022e9704801","Type":"ContainerStarted","Data":"6266d96251ad1f9a773294f581a8eb5c38b75df4dc02e9c098232fee982d43be"} Nov 02 18:15:03.105667 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.105516 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" event={"ID":"bd2ceb92-645d-4255-89a8-c0211bc08025","Type":"ContainerStarted","Data":"131a871760b35b17f7692044161936ce75f7ee543039c887f0ad7b7cfb006e24"} Nov 02 18:15:03.107496 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.107468 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s" event={"ID":"aa194666-bb2c-4610-a051-7fd9caef77d3","Type":"ContainerStarted","Data":"71846bfab522d5f51d3a14b04387e76613f410230c6bed3b9d2ca576f99ad481"} Nov 02 18:15:03.108881 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.108856 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg" event={"ID":"3611e836-f87d-448f-8f33-5f58724e18cf","Type":"ContainerStarted","Data":"7fc6ae65a3d51b510a50eec6d4eaa3f30e9bce1352afb23233d421a5512d3377"} Nov 02 18:15:03.110720 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.110696 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" event={"ID":"e2f91fe1-6e31-4c8c-a24e-2d473a179960","Type":"ContainerStarted","Data":"7c1541fdc8df4beeffeddd3b9411e29a9d0217b02e6efaf37a0a9e933a5b67bd"} Nov 02 18:15:03.112394 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.112372 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j" event={"ID":"6612ae11-596f-4056-b6bc-42ed18333072","Type":"ContainerStarted","Data":"c93399e5267cf3e3153b482aa16b77740f472b17beb07049ff9accc567245355"} Nov 02 18:15:03.117989 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.117942 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z" podStartSLOduration=3.11792547 podStartE2EDuration="3.11792547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.116436899 +0000 UTC m=+2467.807490914" watchObservedRunningTime="2026-11-02 18:15:03.11792547 +0000 UTC m=+2467.808979481" Nov 02 18:15:03.151898 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.151829 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt" podStartSLOduration=3.151806914 podStartE2EDuration="3.151806914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.136305617 +0000 UTC m=+2467.827359629" watchObservedRunningTime="2026-11-02 18:15:03.151806914 +0000 UTC m=+2467.842860929" Nov 02 18:15:03.158052 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.157992 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s" podStartSLOduration=3.157972925 podStartE2EDuration="3.157972925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.155980523 +0000 UTC m=+2467.847034540" watchObservedRunningTime="2026-11-02 18:15:03.157972925 +0000 UTC m=+2467.849026941" Nov 02 18:15:03.183872 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.183807 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j" podStartSLOduration=3.183788064 podStartE2EDuration="3.183788064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.177606826 +0000 UTC m=+2467.868660838" watchObservedRunningTime="2026-11-02 18:15:03.183788064 +0000 UTC m=+2467.874842085" Nov 02 18:15:03.312723 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.309764 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" podStartSLOduration=3.309738165 podStartE2EDuration="3.309738165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.195254103 +0000 UTC m=+2467.886308115" watchObservedRunningTime="2026-11-02 18:15:03.309738165 +0000 UTC m=+2468.000792177" Nov 02 18:15:03.840343 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:03.839561 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" podUID="1a525fd4-e6e8-4ffa-a902-b89637d8d993" containerName="payload-processing" containerID="cri-o://2dc0a6f6d733eb0bbe322a9f2085e2879254a8a77016417ef46ed709ea952152" gracePeriod=25 Nov 02 18:15:04.318901 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.318711 5601 generic.go:356] "Generic (PLEG): container finished" podID="8d603436-a2a3-41f2-8554-c654ac4a9a45" containerID="b9165dc79af6077d94eafb7e850e87f649f1903f4719bacdeef18de7b10eef95" exitCode=0 Nov 02 18:15:04.318901 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.318813 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" event={"ID":"8d603436-a2a3-41f2-8554-c654ac4a9a45","Type":"ContainerDied","Data":"b9165dc79af6077d94eafb7e850e87f649f1903f4719bacdeef18de7b10eef95"} Nov 02 18:15:04.355797 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.355740 5601 generic.go:356] "Generic (PLEG): container finished" podID="e6aed869-903a-4e44-aaed-66d8940f4f90" containerID="a7a2be62b7efda859aefe70cccdd92eb6d3d5a94d279a823f10d24ea53442687" exitCode=60 Nov 02 18:15:04.355979 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.355929 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z" event={"ID":"e6aed869-903a-4e44-aaed-66d8940f4f90","Type":"ContainerDied","Data":"a7a2be62b7efda859aefe70cccdd92eb6d3d5a94d279a823f10d24ea53442687"} Nov 02 18:15:04.356561 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.356511 5601 scope.go:122] "RemoveContainer" containerID="a7a2be62b7efda859aefe70cccdd92eb6d3d5a94d279a823f10d24ea53442687" Nov 02 18:15:04.362832 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.362246 5601 generic.go:356] "Generic (PLEG): container finished" podID="23af2a44-3074-4dce-a5c6-9c18b4c87d35" containerID="88d7ebfce8cc7f812b4202f39d7d84a03d82677f96c5b46039cfbd9002055fbb" exitCode=60 Nov 02 18:15:04.362832 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.362333 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588" event={"ID":"23af2a44-3074-4dce-a5c6-9c18b4c87d35","Type":"ContainerDied","Data":"88d7ebfce8cc7f812b4202f39d7d84a03d82677f96c5b46039cfbd9002055fbb"} Nov 02 18:15:04.364953 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.364353 5601 scope.go:122] "RemoveContainer" containerID="88d7ebfce8cc7f812b4202f39d7d84a03d82677f96c5b46039cfbd9002055fbb" Nov 02 18:15:04.370641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.369478 5601 generic.go:356] "Generic (PLEG): container finished" podID="154aa781-e85c-4aad-8fc1-e022e9704801" containerID="dd3d38e37961d91273c8ca7451ad8617e8c69bd63daf6788e0b727ee6bbdfe29" exitCode=60 Nov 02 18:15:04.370641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.369606 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt" event={"ID":"154aa781-e85c-4aad-8fc1-e022e9704801","Type":"ContainerDied","Data":"dd3d38e37961d91273c8ca7451ad8617e8c69bd63daf6788e0b727ee6bbdfe29"} Nov 02 18:15:04.370641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.370129 5601 scope.go:122] "RemoveContainer" containerID="dd3d38e37961d91273c8ca7451ad8617e8c69bd63daf6788e0b727ee6bbdfe29" Nov 02 18:15:04.384360 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.383349 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" event={"ID":"bd2ceb92-645d-4255-89a8-c0211bc08025","Type":"ContainerStarted","Data":"a617cfe2c89977a22fb813aee0c665171e4bc6b3391c03ccc28ec0eac6a565c1"} Nov 02 18:15:04.394489 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.393755 5601 generic.go:356] "Generic (PLEG): container finished" podID="aa194666-bb2c-4610-a051-7fd9caef77d3" containerID="fc86dc04699e381711cfd5ece21425d5ecd2f0d948fa04b0ec64a5bf3732c67b" exitCode=60 Nov 02 18:15:04.394489 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.394375 5601 scope.go:122] "RemoveContainer" containerID="fc86dc04699e381711cfd5ece21425d5ecd2f0d948fa04b0ec64a5bf3732c67b" Nov 02 18:15:04.394489 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.394418 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s" event={"ID":"aa194666-bb2c-4610-a051-7fd9caef77d3","Type":"ContainerDied","Data":"fc86dc04699e381711cfd5ece21425d5ecd2f0d948fa04b0ec64a5bf3732c67b"} Nov 02 18:15:04.405420 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.404364 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg" event={"ID":"3611e836-f87d-448f-8f33-5f58724e18cf","Type":"ContainerStarted","Data":"4a028e4e6d138c9515be4f2af10eb0923bdca21d5ba79176b82fcaa283827514"} Nov 02 18:15:04.417333 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.417279 5601 generic.go:356] "Generic (PLEG): container finished" podID="e2f91fe1-6e31-4c8c-a24e-2d473a179960" containerID="7c1541fdc8df4beeffeddd3b9411e29a9d0217b02e6efaf37a0a9e933a5b67bd" exitCode=60 Nov 02 18:15:04.419039 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.419009 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" event={"ID":"e2f91fe1-6e31-4c8c-a24e-2d473a179960","Type":"ContainerDied","Data":"7c1541fdc8df4beeffeddd3b9411e29a9d0217b02e6efaf37a0a9e933a5b67bd"} Nov 02 18:15:04.420582 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.420558 5601 scope.go:122] "RemoveContainer" containerID="7c1541fdc8df4beeffeddd3b9411e29a9d0217b02e6efaf37a0a9e933a5b67bd" Nov 02 18:15:04.424174 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.424145 5601 generic.go:356] "Generic (PLEG): container finished" podID="6612ae11-596f-4056-b6bc-42ed18333072" containerID="e036b37a7eb134cc58fd2439dd3c1376d00587ae485f570a273b39d4dbe0bc69" exitCode=60 Nov 02 18:15:04.424435 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.424408 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j" event={"ID":"6612ae11-596f-4056-b6bc-42ed18333072","Type":"ContainerDied","Data":"e036b37a7eb134cc58fd2439dd3c1376d00587ae485f570a273b39d4dbe0bc69"} Nov 02 18:15:04.425193 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.425151 5601 scope.go:122] "RemoveContainer" containerID="e036b37a7eb134cc58fd2439dd3c1376d00587ae485f570a273b39d4dbe0bc69" Nov 02 18:15:04.525177 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.427919 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" podUID="46c9a1a3-ff28-4d6c-9d85-faa3be401911" containerName="payload-processing" containerID="cri-o://9c9cf3b65b791c23410e500f1ac0313ab7bc27bf3d751b480fbeee12de5bc7fd" gracePeriod=25 Nov 02 18:15:04.525177 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.432329 5601 generic.go:356] "Generic (PLEG): container finished" podID="1a525fd4-e6e8-4ffa-a902-b89637d8d993" containerID="2dc0a6f6d733eb0bbe322a9f2085e2879254a8a77016417ef46ed709ea952152" exitCode=0 Nov 02 18:15:04.525177 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.432360 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" event={"ID":"1a525fd4-e6e8-4ffa-a902-b89637d8d993","Type":"ContainerDied","Data":"2dc0a6f6d733eb0bbe322a9f2085e2879254a8a77016417ef46ed709ea952152"} Nov 02 18:15:04.548307 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.547025 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" event={"ID":"c4e45226-6362-4b36-b890-11b6804c8d65","Type":"ContainerStarted","Data":"32529cc757a1be93c0565d2e498efc33d9286fbdefbc09f77219185bba8e8d24"} Nov 02 18:15:04.553364 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.553099 5601 generic.go:356] "Generic (PLEG): container finished" podID="b16fa692-4b98-42ad-b4db-b38d411c093a" containerID="941456ebe919c3ca96ecd4cfec08ee695bd195e45d582098e84ba3e7ee03abf1" exitCode=60 Nov 02 18:15:04.553364 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.553191 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r" event={"ID":"b16fa692-4b98-42ad-b4db-b38d411c093a","Type":"ContainerDied","Data":"941456ebe919c3ca96ecd4cfec08ee695bd195e45d582098e84ba3e7ee03abf1"} Nov 02 18:15:04.554521 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.554003 5601 scope.go:122] "RemoveContainer" containerID="941456ebe919c3ca96ecd4cfec08ee695bd195e45d582098e84ba3e7ee03abf1" Nov 02 18:15:04.587982 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.587883 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg" podStartSLOduration=4.587860545 podStartE2EDuration="4.587860545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:04.561031261 +0000 UTC m=+2469.252085274" watchObservedRunningTime="2026-11-02 18:15:04.587860545 +0000 UTC m=+2469.278914559" Nov 02 18:15:04.623071 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:04.622487 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:15:04.625645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.624817 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" event={"ID":"3e296182-a14a-4ee7-9cd2-d014826b6cd9","Type":"ContainerStarted","Data":"57632db5054ee6c4f08dbcf0f40865c32a03d531f05f7ae57bfc618c3aa7cb86"} Nov 02 18:15:04.637917 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.637846 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" podStartSLOduration=4.637825186 podStartE2EDuration="4.637825186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:04.590682284 +0000 UTC m=+2469.281736297" watchObservedRunningTime="2026-11-02 18:15:04.637825186 +0000 UTC m=+2469.328879199" Nov 02 18:15:04.686673 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.682863 5601 generic.go:356] "Generic (PLEG): container finished" podID="62588dd0-b7ee-440e-ab7e-419d7429b2d9" containerID="a78f6dab9abdfd2cd41b6ef29f40eb9b0ae45053693fbaf3d7b0ff40afcf53ab" exitCode=60 Nov 02 18:15:04.686673 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.682990 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb" event={"ID":"62588dd0-b7ee-440e-ab7e-419d7429b2d9","Type":"ContainerDied","Data":"a78f6dab9abdfd2cd41b6ef29f40eb9b0ae45053693fbaf3d7b0ff40afcf53ab"} Nov 02 18:15:04.686673 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.683960 5601 scope.go:122] "RemoveContainer" containerID="a78f6dab9abdfd2cd41b6ef29f40eb9b0ae45053693fbaf3d7b0ff40afcf53ab" Nov 02 18:15:04.714013 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.712962 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" podStartSLOduration=4.712942039 podStartE2EDuration="4.712942039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:04.70936772 +0000 UTC m=+2469.400421733" watchObservedRunningTime="2026-11-02 18:15:04.712942039 +0000 UTC m=+2469.403996057" Nov 02 18:15:04.727029 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.726993 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" event={"ID":"871bed77-cbe7-4001-a4f8-a6cedca05cef","Type":"ContainerStarted","Data":"ec4033eecceea7655a82e4ff13126484df0a56fae0a494d3b75bb6496a8644f7"} Nov 02 18:15:04.763960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.752251 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" podStartSLOduration=4.752230809 podStartE2EDuration="4.752230809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:04.729882856 +0000 UTC m=+2469.420936873" watchObservedRunningTime="2026-11-02 18:15:04.752230809 +0000 UTC m=+2469.443284824" Nov 02 18:15:04.818907 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:04.818836 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" podStartSLOduration=4.818814531 podStartE2EDuration="4.818814531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:04.786213523 +0000 UTC m=+2469.477267535" watchObservedRunningTime="2026-11-02 18:15:04.818814531 +0000 UTC m=+2469.509868544" Nov 02 18:15:05.108027 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.107008 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:15:05.119001 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.118049 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" Nov 02 18:15:05.329529 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.323292 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-credential-socket\") pod \"8d603436-a2a3-41f2-8554-c654ac4a9a45\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " Nov 02 18:15:05.329529 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.323397 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-workload-socket\") pod \"8d603436-a2a3-41f2-8554-c654ac4a9a45\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " Nov 02 18:15:05.329529 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.323482 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-data\") pod \"8d603436-a2a3-41f2-8554-c654ac4a9a45\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " Nov 02 18:15:05.329529 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.323575 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-workload-certs\") pod \"8d603436-a2a3-41f2-8554-c654ac4a9a45\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " Nov 02 18:15:05.329529 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.323676 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a525fd4-e6e8-4ffa-a902-b89637d8d993-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a525fd4-e6e8-4ffa-a902-b89637d8d993-plugins-config-volume\") pod \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\" (UID: \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\") " Nov 02 18:15:05.329529 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.323739 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a525fd4-e6e8-4ffa-a902-b89637d8d993-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a525fd4-e6e8-4ffa-a902-b89637d8d993-routing-overlay-volume\") pod \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\" (UID: \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\") " Nov 02 18:15:05.329529 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.323796 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-podinfo\") pod \"8d603436-a2a3-41f2-8554-c654ac4a9a45\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " Nov 02 18:15:05.329529 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.323905 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d603436-a2a3-41f2-8554-c654ac4a9a45-kube-api-access-594ql\" (UniqueName: \"kubernetes.io/projected/8d603436-a2a3-41f2-8554-c654ac4a9a45-kube-api-access-594ql\") pod \"8d603436-a2a3-41f2-8554-c654ac4a9a45\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " Nov 02 18:15:05.329529 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.323948 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-token\" (UniqueName: \"kubernetes.io/projected/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-token\") pod \"8d603436-a2a3-41f2-8554-c654ac4a9a45\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " Nov 02 18:15:05.329529 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.323999 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a525fd4-e6e8-4ffa-a902-b89637d8d993-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a525fd4-e6e8-4ffa-a902-b89637d8d993-provider-credentials\") pod \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\" (UID: \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\") " Nov 02 18:15:05.329529 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.324073 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d603436-a2a3-41f2-8554-c654ac4a9a45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d603436-a2a3-41f2-8554-c654ac4a9a45-istiod-ca-cert\") pod \"8d603436-a2a3-41f2-8554-c654ac4a9a45\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " Nov 02 18:15:05.329529 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.324164 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a525fd4-e6e8-4ffa-a902-b89637d8d993-kube-api-access-252f2\" (UniqueName: \"kubernetes.io/projected/1a525fd4-e6e8-4ffa-a902-b89637d8d993-kube-api-access-252f2\") pod \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\" (UID: \"1a525fd4-e6e8-4ffa-a902-b89637d8d993\") " Nov 02 18:15:05.329529 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.324230 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-envoy\") pod \"8d603436-a2a3-41f2-8554-c654ac4a9a45\" (UID: \"8d603436-a2a3-41f2-8554-c654ac4a9a45\") " Nov 02 18:15:05.329529 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.327053 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a525fd4-e6e8-4ffa-a902-b89637d8d993-routing-overlay-volume" pod "1a525fd4-e6e8-4ffa-a902-b89637d8d993" (UID: "1a525fd4-e6e8-4ffa-a902-b89637d8d993"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:05.329529 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.328169 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-credential-socket" pod "8d603436-a2a3-41f2-8554-c654ac4a9a45" (UID: "8d603436-a2a3-41f2-8554-c654ac4a9a45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:05.335179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.331846 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-workload-socket" pod "8d603436-a2a3-41f2-8554-c654ac4a9a45" (UID: "8d603436-a2a3-41f2-8554-c654ac4a9a45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:05.335179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.332171 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-data" pod "8d603436-a2a3-41f2-8554-c654ac4a9a45" (UID: "8d603436-a2a3-41f2-8554-c654ac4a9a45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:05.335179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.333279 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-workload-certs" pod "8d603436-a2a3-41f2-8554-c654ac4a9a45" (UID: "8d603436-a2a3-41f2-8554-c654ac4a9a45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:05.335179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.333761 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a525fd4-e6e8-4ffa-a902-b89637d8d993-plugins-config-volume" pod "1a525fd4-e6e8-4ffa-a902-b89637d8d993" (UID: "1a525fd4-e6e8-4ffa-a902-b89637d8d993"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:05.729033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.554667 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d603436-a2a3-41f2-8554-c654ac4a9a45-istiod-ca-cert" pod "8d603436-a2a3-41f2-8554-c654ac4a9a45" (UID: "8d603436-a2a3-41f2-8554-c654ac4a9a45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:05.729033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.664844 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d603436-a2a3-41f2-8554-c654ac4a9a45-kube-api-access-594ql" pod "8d603436-a2a3-41f2-8554-c654ac4a9a45" (UID: "8d603436-a2a3-41f2-8554-c654ac4a9a45"). InnerVolumeSpecName "kube-api-access-594ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:05.729033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.703085 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-token" pod "8d603436-a2a3-41f2-8554-c654ac4a9a45" (UID: "8d603436-a2a3-41f2-8554-c654ac4a9a45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:05.729033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.707671 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:05.729033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.707732 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:05.729033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.707752 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:05.729033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.707789 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:05.729033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.707807 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a525fd4-e6e8-4ffa-a902-b89637d8d993-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:05.729033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.707823 5601 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a525fd4-e6e8-4ffa-a902-b89637d8d993-routing-overlay-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:05.729033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.708113 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-594ql\" (UniqueName: \"kubernetes.io/projected/8d603436-a2a3-41f2-8554-c654ac4a9a45-kube-api-access-594ql\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:05.729033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.708141 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d603436-a2a3-41f2-8554-c654ac4a9a45-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:05.794642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.786257 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a525fd4-e6e8-4ffa-a902-b89637d8d993-provider-credentials" pod "1a525fd4-e6e8-4ffa-a902-b89637d8d993" (UID: "1a525fd4-e6e8-4ffa-a902-b89637d8d993"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:05.794642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.787935 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-podinfo" pod "8d603436-a2a3-41f2-8554-c654ac4a9a45" (UID: "8d603436-a2a3-41f2-8554-c654ac4a9a45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:15:05.795332 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.795105 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-envoy" pod "8d603436-a2a3-41f2-8554-c654ac4a9a45" (UID: "8d603436-a2a3-41f2-8554-c654ac4a9a45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:05.819852 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.819281 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:05.819852 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.819319 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:05.819852 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.819343 5601 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a525fd4-e6e8-4ffa-a902-b89637d8d993-provider-credentials\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:05.819852 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.819360 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d603436-a2a3-41f2-8554-c654ac4a9a45-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:05.926997 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:05.926921 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a525fd4-e6e8-4ffa-a902-b89637d8d993-kube-api-access-252f2" pod "1a525fd4-e6e8-4ffa-a902-b89637d8d993" (UID: "1a525fd4-e6e8-4ffa-a902-b89637d8d993"). InnerVolumeSpecName "kube-api-access-252f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:06.038639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.034749 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-252f2\" (UniqueName: \"kubernetes.io/projected/1a525fd4-e6e8-4ffa-a902-b89637d8d993-kube-api-access-252f2\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:06.165037 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.164986 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" Nov 02 18:15:06.243077 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.227861 5601 generic.go:356] "Generic (PLEG): container finished" podID="3e296182-a14a-4ee7-9cd2-d014826b6cd9" containerID="57632db5054ee6c4f08dbcf0f40865c32a03d531f05f7ae57bfc618c3aa7cb86" exitCode=60 Nov 02 18:15:06.243077 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.227982 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" event={"ID":"3e296182-a14a-4ee7-9cd2-d014826b6cd9","Type":"ContainerDied","Data":"57632db5054ee6c4f08dbcf0f40865c32a03d531f05f7ae57bfc618c3aa7cb86"} Nov 02 18:15:06.243077 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.228544 5601 scope.go:122] "RemoveContainer" containerID="57632db5054ee6c4f08dbcf0f40865c32a03d531f05f7ae57bfc618c3aa7cb86" Nov 02 18:15:06.243077 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.242986 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c9a1a3-ff28-4d6c-9d85-faa3be401911-kube-api-access-jt6wr\" (UniqueName: \"kubernetes.io/projected/46c9a1a3-ff28-4d6c-9d85-faa3be401911-kube-api-access-jt6wr\") pod \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\" (UID: \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\") " Nov 02 18:15:06.243455 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.243116 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46c9a1a3-ff28-4d6c-9d85-faa3be401911-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c9a1a3-ff28-4d6c-9d85-faa3be401911-plugins-config-volume\") pod \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\" (UID: \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\") " Nov 02 18:15:06.243455 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.243159 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c9a1a3-ff28-4d6c-9d85-faa3be401911-provider-credentials\" (UniqueName: \"kubernetes.io/projected/46c9a1a3-ff28-4d6c-9d85-faa3be401911-provider-credentials\") pod \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\" (UID: \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\") " Nov 02 18:15:06.243575 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.243459 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46c9a1a3-ff28-4d6c-9d85-faa3be401911-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46c9a1a3-ff28-4d6c-9d85-faa3be401911-routing-overlay-volume\") pod \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\" (UID: \"46c9a1a3-ff28-4d6c-9d85-faa3be401911\") " Nov 02 18:15:06.251422 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.251383 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c9a1a3-ff28-4d6c-9d85-faa3be401911-plugins-config-volume" pod "46c9a1a3-ff28-4d6c-9d85-faa3be401911" (UID: "46c9a1a3-ff28-4d6c-9d85-faa3be401911"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:06.307961 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.302374 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c9a1a3-ff28-4d6c-9d85-faa3be401911-provider-credentials" pod "46c9a1a3-ff28-4d6c-9d85-faa3be401911" (UID: "46c9a1a3-ff28-4d6c-9d85-faa3be401911"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:06.352730 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.315287 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c9a1a3-ff28-4d6c-9d85-faa3be401911-kube-api-access-jt6wr" pod "46c9a1a3-ff28-4d6c-9d85-faa3be401911" (UID: "46c9a1a3-ff28-4d6c-9d85-faa3be401911"). InnerVolumeSpecName "kube-api-access-jt6wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:06.353310 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:15:06.352087 5601 logging.go:55] [core] [Channel #1829 SubChannel #1830]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.158:50052", ServerName: "10.134.0.158:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.158:50052: operation was canceled" Nov 02 18:15:06.353433 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.349534 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c9a1a3-ff28-4d6c-9d85-faa3be401911-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:06.353528 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.351999 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" podUID="46c9a1a3-ff28-4d6c-9d85-faa3be401911" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.158:50052\" within 1s: context deadline exceeded" Nov 02 18:15:06.357419 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.354118 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c9a1a3-ff28-4d6c-9d85-faa3be401911-routing-overlay-volume" pod "46c9a1a3-ff28-4d6c-9d85-faa3be401911" (UID: "46c9a1a3-ff28-4d6c-9d85-faa3be401911"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:06.499596 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.485040 5601 generic.go:356] "Generic (PLEG): container finished" podID="871bed77-cbe7-4001-a4f8-a6cedca05cef" containerID="ec4033eecceea7655a82e4ff13126484df0a56fae0a494d3b75bb6496a8644f7" exitCode=60 Nov 02 18:15:06.499596 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.485235 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" event={"ID":"871bed77-cbe7-4001-a4f8-a6cedca05cef","Type":"ContainerDied","Data":"ec4033eecceea7655a82e4ff13126484df0a56fae0a494d3b75bb6496a8644f7"} Nov 02 18:15:06.499596 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.498995 5601 scope.go:122] "RemoveContainer" containerID="ec4033eecceea7655a82e4ff13126484df0a56fae0a494d3b75bb6496a8644f7" Nov 02 18:15:06.559641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.502174 5601 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46c9a1a3-ff28-4d6c-9d85-faa3be401911-provider-credentials\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:06.571676 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.561403 5601 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46c9a1a3-ff28-4d6c-9d85-faa3be401911-routing-overlay-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:06.571676 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.561438 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt6wr\" (UniqueName: \"kubernetes.io/projected/46c9a1a3-ff28-4d6c-9d85-faa3be401911-kube-api-access-jt6wr\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:06.571676 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.519148 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" Nov 02 18:15:06.571676 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.519049 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf" event={"ID":"8d603436-a2a3-41f2-8554-c654ac4a9a45","Type":"ContainerDied","Data":"0c28148396e659f3e7f4ef032063840b858faddfac72b1823635223756ba9d76"} Nov 02 18:15:06.571676 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.562433 5601 scope.go:122] "RemoveContainer" containerID="b9165dc79af6077d94eafb7e850e87f649f1903f4719bacdeef18de7b10eef95" Nov 02 18:15:06.715458 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.715418 5601 generic.go:356] "Generic (PLEG): container finished" podID="bd2ceb92-645d-4255-89a8-c0211bc08025" containerID="a617cfe2c89977a22fb813aee0c665171e4bc6b3391c03ccc28ec0eac6a565c1" exitCode=60 Nov 02 18:15:06.720423 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.720395 5601 scope.go:122] "RemoveContainer" containerID="a617cfe2c89977a22fb813aee0c665171e4bc6b3391c03ccc28ec0eac6a565c1" Nov 02 18:15:06.721750 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.721722 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" event={"ID":"bd2ceb92-645d-4255-89a8-c0211bc08025","Type":"ContainerDied","Data":"a617cfe2c89977a22fb813aee0c665171e4bc6b3391c03ccc28ec0eac6a565c1"} Nov 02 18:15:06.723741 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.723650 5601 generic.go:356] "Generic (PLEG): container finished" podID="3611e836-f87d-448f-8f33-5f58724e18cf" containerID="4a028e4e6d138c9515be4f2af10eb0923bdca21d5ba79176b82fcaa283827514" exitCode=60 Nov 02 18:15:06.723996 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.723873 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg" event={"ID":"3611e836-f87d-448f-8f33-5f58724e18cf","Type":"ContainerDied","Data":"4a028e4e6d138c9515be4f2af10eb0923bdca21d5ba79176b82fcaa283827514"} Nov 02 18:15:06.726826 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.726806 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" event={"ID":"46c9a1a3-ff28-4d6c-9d85-faa3be401911","Type":"ContainerDied","Data":"9c9cf3b65b791c23410e500f1ac0313ab7bc27bf3d751b480fbeee12de5bc7fd"} Nov 02 18:15:06.727001 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.726596 5601 generic.go:356] "Generic (PLEG): container finished" podID="46c9a1a3-ff28-4d6c-9d85-faa3be401911" containerID="9c9cf3b65b791c23410e500f1ac0313ab7bc27bf3d751b480fbeee12de5bc7fd" exitCode=0 Nov 02 18:15:06.727156 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.724065 5601 scope.go:122] "RemoveContainer" containerID="4a028e4e6d138c9515be4f2af10eb0923bdca21d5ba79176b82fcaa283827514" Nov 02 18:15:06.735454 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.726672 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl" Nov 02 18:15:06.774004 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.773973 5601 scope.go:122] "RemoveContainer" containerID="9c9cf3b65b791c23410e500f1ac0313ab7bc27bf3d751b480fbeee12de5bc7fd" Nov 02 18:15:06.903734 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.883142 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" event={"ID":"1a525fd4-e6e8-4ffa-a902-b89637d8d993","Type":"ContainerDied","Data":"0277bc133e2d453d0470c423d03982b6d05661581a2ed3a5598cf0e28dc07300"} Nov 02 18:15:06.903734 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.883287 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2" Nov 02 18:15:07.052361 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.970811 5601 generic.go:356] "Generic (PLEG): container finished" podID="18f2b645-9cdf-4b3e-8b54-8e31706d33a3" containerID="0aa40b01a447acb7b0cb45f8a6ee09e931fdbc447890430266c3cb0f6f447096" exitCode=60 Nov 02 18:15:07.052361 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.970921 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" event={"ID":"18f2b645-9cdf-4b3e-8b54-8e31706d33a3","Type":"ContainerDied","Data":"0aa40b01a447acb7b0cb45f8a6ee09e931fdbc447890430266c3cb0f6f447096"} Nov 02 18:15:07.052361 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:06.971490 5601 scope.go:122] "RemoveContainer" containerID="0aa40b01a447acb7b0cb45f8a6ee09e931fdbc447890430266c3cb0f6f447096" Nov 02 18:15:07.052361 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:06.972416 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf_odh-ai-gateway-infra(18f2b645-9cdf-4b3e-8b54-8e31706d33a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" podUID="18f2b645-9cdf-4b3e-8b54-8e31706d33a3" Nov 02 18:15:07.193677 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:07.177684 5601 generic.go:356] "Generic (PLEG): container finished" podID="c4e45226-6362-4b36-b890-11b6804c8d65" containerID="32529cc757a1be93c0565d2e498efc33d9286fbdefbc09f77219185bba8e8d24" exitCode=60 Nov 02 18:15:07.193677 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:07.177765 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" event={"ID":"c4e45226-6362-4b36-b890-11b6804c8d65","Type":"ContainerDied","Data":"32529cc757a1be93c0565d2e498efc33d9286fbdefbc09f77219185bba8e8d24"} Nov 02 18:15:07.207026 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:07.206989 5601 scope.go:122] "RemoveContainer" containerID="32529cc757a1be93c0565d2e498efc33d9286fbdefbc09f77219185bba8e8d24" Nov 02 18:15:07.430439 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:07.420296 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf"] Nov 02 18:15:07.603916 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:07.603820 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 18:15:07.637548 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:07.637495 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:15:07.679909 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:07.679867 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:15:07.681475 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:07.681441 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 18:15:07.706003 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:07.690497 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 18:15:07.871696 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:07.861786 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 18:15:07.886710 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:07.882512 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-rdssf"] Nov 02 18:15:07.934137 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:07.934089 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:15:07.995944 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:07.995908 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml"] Nov 02 18:15:07.996477 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:07.996437 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" podUID="17487351-07f6-4585-aa0c-80766361b815" containerName="istio-proxy" containerID="cri-o://e9c373df4090677e381f08144a011f7b05d4224e756b54d22fa52275cae1f35b" gracePeriod=30 Nov 02 18:15:08.069504 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:08.069019 5601 scope.go:122] "RemoveContainer" containerID="2dc0a6f6d733eb0bbe322a9f2085e2879254a8a77016417ef46ed709ea952152" Nov 02 18:15:08.169696 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:08.162018 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 18:15:08.187282 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:08.184773 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:15:08.196767 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:08.194650 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj"] Nov 02 18:15:08.196767 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:08.195473 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" podUID="b7d8a50a-572b-48b7-8c69-93d75c6c78eb" containerName="istio-proxy" containerID="cri-o://857937fb5820e416f86f7cc96de9ac48e12f4a5d632bbd6ca10320a6ce9c748c" gracePeriod=30 Nov 02 18:15:08.324071 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:08.322887 5601 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a525fd4_e6e8_4ffa_a902_b89637d8d993.slice/crio-0277bc133e2d453d0470c423d03982b6d05661581a2ed3a5598cf0e28dc07300\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a525fd4_e6e8_4ffa_a902_b89637d8d993.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6aed869_903a_4e44_aaed_66d8940f4f90.slice/crio-conmon-cde0423afcc4c2495be038a4819183dfced23f0e623f175b645d61a806897537.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa194666_bb2c_4610_a051_7fd9caef77d3.slice/crio-conmon-419216f1f5f94e112f2438bf352d996c57e38f8a03253b24eddcf2fc7eb28b4c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23af2a44_3074_4dce_a5c6_9c18b4c87d35.slice/crio-91d1550e90725e58625868b61b5917a8d3278b510611ad58bf1390a88918704d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa194666_bb2c_4610_a051_7fd9caef77d3.slice/crio-419216f1f5f94e112f2438bf352d996c57e38f8a03253b24eddcf2fc7eb28b4c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23af2a44_3074_4dce_a5c6_9c18b4c87d35.slice/crio-conmon-91d1550e90725e58625868b61b5917a8d3278b510611ad58bf1390a88918704d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154aa781_e85c_4aad_8fc1_e022e9704801.slice/crio-conmon-0649a8c60b016d81ac348477987b182bc6ee6d2e8aaf30f97d41c2f30e5f2882.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c9a1a3_ff28_4d6c_9d85_faa3be401911.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154aa781_e85c_4aad_8fc1_e022e9704801.slice/crio-0649a8c60b016d81ac348477987b182bc6ee6d2e8aaf30f97d41c2f30e5f2882.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6612ae11_596f_4056_b6bc_42ed18333072.slice/crio-conmon-d93a689c9cfa4fe350d2c16a976c8835fdb77ccd8942a8770685635bb2de2939.scope\": RecentStats: unable to find data in memory cache]" Nov 02 18:15:08.325317 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:08.325270 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:15:08.349961 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:08.349931 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2"] Nov 02 18:15:08.632522 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:08.594222 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:15:08.708283 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:08.703240 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded" Nov 02 18:15:08.745659 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:08.719081 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 18:15:08.745659 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:08.719585 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 18:15:08.745659 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:08.720458 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 18:15:08.745659 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:08.720871 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 18:15:08.810651 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:08.761120 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Nov 02 18:15:08.810651 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:08.761647 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 18:15:08.810651 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:08.762160 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 18:15:08.810651 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:08.762529 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 18:15:08.904996 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:08.904954 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 18:15:08.905869 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:08.905832 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 18:15:09.218762 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:09.218706 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 02 18:15:09.219212 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:09.219185 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 18:15:09.455762 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:09.455695 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:15:09.461638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:09.456641 5601 scope.go:122] "RemoveContainer" containerID="382f729da268d7107a5af93ae8380488924896aa24ef37069a42b16224b6fa09" Nov 02 18:15:09.625284 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:09.625214 5601 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3611e836_f87d_448f_8f33_5f58724e18cf.slice/crio-ec824a2722e7687d4acf37ef7f3ff0eaf7dea94d18a7e15d6f33e7a68fa734e7.scope\": RecentStats: unable to find data in memory cache]" Nov 02 18:15:09.632095 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:09.631665 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:15:09.671645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:09.666304 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d603436-a2a3-41f2-8554-c654ac4a9a45" path="/var/lib/kubelet/pods/8d603436-a2a3-41f2-8554-c654ac4a9a45/volumes" Nov 02 18:15:09.720152 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:09.720114 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 18:15:09.720405 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:09.720387 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-c7cf7zjc2"] Nov 02 18:15:09.720753 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:09.720733 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 18:15:09.720987 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:09.720971 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 18:15:09.722018 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:09.721942 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 18:15:09.726345 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:09.722194 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 18:15:09.741480 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:09.741445 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl"] Nov 02 18:15:09.801836 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:09.775752 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-6f65f75cl"] Nov 02 18:15:09.801836 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:09.775845 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 18:15:09.801836 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:09.775889 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 18:15:09.801836 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:09.775924 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 18:15:09.943693 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:09.928823 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 18:15:10.117779 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.105541 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 18:15:10.202862 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.202767 5601 generic.go:356] "Generic (PLEG): container finished" podID="e6aed869-903a-4e44-aaed-66d8940f4f90" containerID="cde0423afcc4c2495be038a4819183dfced23f0e623f175b645d61a806897537" exitCode=60 Nov 02 18:15:10.203159 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.203134 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z" event={"ID":"e6aed869-903a-4e44-aaed-66d8940f4f90","Type":"ContainerDied","Data":"cde0423afcc4c2495be038a4819183dfced23f0e623f175b645d61a806897537"} Nov 02 18:15:10.203303 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.203289 5601 scope.go:122] "RemoveContainer" containerID="a7a2be62b7efda859aefe70cccdd92eb6d3d5a94d279a823f10d24ea53442687" Nov 02 18:15:10.211894 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.211863 5601 scope.go:122] "RemoveContainer" containerID="cde0423afcc4c2495be038a4819183dfced23f0e623f175b645d61a806897537" Nov 02 18:15:10.212740 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:10.212600 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z_odh-ai-gateway-infra(e6aed869-903a-4e44-aaed-66d8940f4f90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z" podUID="e6aed869-903a-4e44-aaed-66d8940f4f90" Nov 02 18:15:10.243898 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.243851 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz"] Nov 02 18:15:10.245475 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.245359 5601 cpu_manager.go:414] "containerMap: removing container" podUID="1a525fd4-e6e8-4ffa-a902-b89637d8d993" containerName="payload-processing" Nov 02 18:15:10.245752 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.245731 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a525fd4-e6e8-4ffa-a902-b89637d8d993" containerName="payload-processing" Nov 02 18:15:10.245889 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.245878 5601 cpu_manager.go:414] "containerMap: removing container" podUID="46c9a1a3-ff28-4d6c-9d85-faa3be401911" containerName="payload-processing" Nov 02 18:15:10.245976 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.245964 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c9a1a3-ff28-4d6c-9d85-faa3be401911" containerName="payload-processing" Nov 02 18:15:10.246090 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.246081 5601 cpu_manager.go:414] "containerMap: removing container" podUID="8d603436-a2a3-41f2-8554-c654ac4a9a45" containerName="istio-proxy" Nov 02 18:15:10.246162 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.246152 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d603436-a2a3-41f2-8554-c654ac4a9a45" containerName="istio-proxy" Nov 02 18:15:10.246495 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.246479 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d603436-a2a3-41f2-8554-c654ac4a9a45" containerName="istio-proxy" Nov 02 18:15:10.246601 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.246591 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a525fd4-e6e8-4ffa-a902-b89637d8d993" containerName="payload-processing" Nov 02 18:15:10.246722 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.246712 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c9a1a3-ff28-4d6c-9d85-faa3be401911" containerName="payload-processing" Nov 02 18:15:10.291221 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.291187 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz"] Nov 02 18:15:10.291524 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.291507 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.296364 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.294712 5601 generic.go:356] "Generic (PLEG): container finished" podID="23af2a44-3074-4dce-a5c6-9c18b4c87d35" containerID="91d1550e90725e58625868b61b5917a8d3278b510611ad58bf1390a88918704d" exitCode=60 Nov 02 18:15:10.296364 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.294811 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588" event={"ID":"23af2a44-3074-4dce-a5c6-9c18b4c87d35","Type":"ContainerDied","Data":"91d1550e90725e58625868b61b5917a8d3278b510611ad58bf1390a88918704d"} Nov 02 18:15:10.298469 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.297907 5601 scope.go:122] "RemoveContainer" containerID="91d1550e90725e58625868b61b5917a8d3278b510611ad58bf1390a88918704d" Nov 02 18:15:10.298469 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:10.298388 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588_odh-ai-gateway-infra(23af2a44-3074-4dce-a5c6-9c18b4c87d35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588" podUID="23af2a44-3074-4dce-a5c6-9c18b4c87d35" Nov 02 18:15:10.300133 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.299508 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-312f21-openshift-default-dockercfg-plxfc\"" Nov 02 18:15:10.302578 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.302444 5601 generic.go:356] "Generic (PLEG): container finished" podID="154aa781-e85c-4aad-8fc1-e022e9704801" containerID="0649a8c60b016d81ac348477987b182bc6ee6d2e8aaf30f97d41c2f30e5f2882" exitCode=60 Nov 02 18:15:10.302794 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.302583 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt" event={"ID":"154aa781-e85c-4aad-8fc1-e022e9704801","Type":"ContainerDied","Data":"0649a8c60b016d81ac348477987b182bc6ee6d2e8aaf30f97d41c2f30e5f2882"} Nov 02 18:15:10.303248 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.303125 5601 scope.go:122] "RemoveContainer" containerID="0649a8c60b016d81ac348477987b182bc6ee6d2e8aaf30f97d41c2f30e5f2882" Nov 02 18:15:10.303731 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:10.303581 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt_odh-ai-gateway-infra(154aa781-e85c-4aad-8fc1-e022e9704801)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt" podUID="154aa781-e85c-4aad-8fc1-e022e9704801" Nov 02 18:15:10.310045 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.310012 5601 generic.go:356] "Generic (PLEG): container finished" podID="aa194666-bb2c-4610-a051-7fd9caef77d3" containerID="419216f1f5f94e112f2438bf352d996c57e38f8a03253b24eddcf2fc7eb28b4c" exitCode=60 Nov 02 18:15:10.311383 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.310834 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s" event={"ID":"aa194666-bb2c-4610-a051-7fd9caef77d3","Type":"ContainerDied","Data":"419216f1f5f94e112f2438bf352d996c57e38f8a03253b24eddcf2fc7eb28b4c"} Nov 02 18:15:10.312972 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.312950 5601 scope.go:122] "RemoveContainer" containerID="419216f1f5f94e112f2438bf352d996c57e38f8a03253b24eddcf2fc7eb28b4c" Nov 02 18:15:10.314451 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:10.314418 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s_odh-ai-gateway-infra(aa194666-bb2c-4610-a051-7fd9caef77d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s" podUID="aa194666-bb2c-4610-a051-7fd9caef77d3" Nov 02 18:15:10.323452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.323425 5601 scope.go:122] "RemoveContainer" containerID="88d7ebfce8cc7f812b4202f39d7d84a03d82677f96c5b46039cfbd9002055fbb" Nov 02 18:15:10.389104 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.389063 5601 generic.go:356] "Generic (PLEG): container finished" podID="e2f91fe1-6e31-4c8c-a24e-2d473a179960" containerID="e434a331d257538659029e4af02c7db96af646b3599b00fe5754f909ae094b89" exitCode=60 Nov 02 18:15:10.397430 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.390468 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-workload-certs\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.398340 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.398312 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945a032d-c69f-4d9e-96a5-db1ac483af67-istiod-ca-cert\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.398682 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.396593 5601 scope.go:122] "RemoveContainer" containerID="e434a331d257538659029e4af02c7db96af646b3599b00fe5754f909ae094b89" Nov 02 18:15:10.398789 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.398607 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-podinfo\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.398951 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.398933 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjdck\" (UniqueName: \"kubernetes.io/projected/945a032d-c69f-4d9e-96a5-db1ac483af67-kube-api-access-zjdck\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.399080 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.399065 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-envoy\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.399208 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.399195 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-token\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.399341 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:10.399301 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj_odh-ai-gateway-infra(e2f91fe1-6e31-4c8c-a24e-2d473a179960)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" podUID="e2f91fe1-6e31-4c8c-a24e-2d473a179960" Nov 02 18:15:10.399437 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.395694 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" event={"ID":"e2f91fe1-6e31-4c8c-a24e-2d473a179960","Type":"ContainerDied","Data":"e434a331d257538659029e4af02c7db96af646b3599b00fe5754f909ae094b89"} Nov 02 18:15:10.400046 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.400016 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-workload-socket\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.400206 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.400085 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-data\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.400206 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.400151 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-credential-socket\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.406787 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.406748 5601 generic.go:356] "Generic (PLEG): container finished" podID="6612ae11-596f-4056-b6bc-42ed18333072" containerID="d93a689c9cfa4fe350d2c16a976c8835fdb77ccd8942a8770685635bb2de2939" exitCode=60 Nov 02 18:15:10.407053 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.406819 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j" event={"ID":"6612ae11-596f-4056-b6bc-42ed18333072","Type":"ContainerDied","Data":"d93a689c9cfa4fe350d2c16a976c8835fdb77ccd8942a8770685635bb2de2939"} Nov 02 18:15:10.407794 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.407773 5601 scope.go:122] "RemoveContainer" containerID="d93a689c9cfa4fe350d2c16a976c8835fdb77ccd8942a8770685635bb2de2939" Nov 02 18:15:10.408451 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:10.408406 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j_odh-ai-gateway-infra(6612ae11-596f-4056-b6bc-42ed18333072)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j" podUID="6612ae11-596f-4056-b6bc-42ed18333072" Nov 02 18:15:10.418694 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.418662 5601 scope.go:122] "RemoveContainer" containerID="0aa40b01a447acb7b0cb45f8a6ee09e931fdbc447890430266c3cb0f6f447096" Nov 02 18:15:10.419187 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:10.419157 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf_odh-ai-gateway-infra(18f2b645-9cdf-4b3e-8b54-8e31706d33a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" podUID="18f2b645-9cdf-4b3e-8b54-8e31706d33a3" Nov 02 18:15:10.421804 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.421772 5601 generic.go:356] "Generic (PLEG): container finished" podID="b16fa692-4b98-42ad-b4db-b38d411c093a" containerID="58726700aa56b2ebb545209bb5ac7a1e0795adc04fc61357d8a97c853ae133fa" exitCode=60 Nov 02 18:15:10.430850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.430813 5601 scope.go:122] "RemoveContainer" containerID="58726700aa56b2ebb545209bb5ac7a1e0795adc04fc61357d8a97c853ae133fa" Nov 02 18:15:10.431178 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.431135 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r" event={"ID":"b16fa692-4b98-42ad-b4db-b38d411c093a","Type":"ContainerDied","Data":"58726700aa56b2ebb545209bb5ac7a1e0795adc04fc61357d8a97c853ae133fa"} Nov 02 18:15:10.431960 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:10.431325 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r_odh-ai-gateway-infra(b16fa692-4b98-42ad-b4db-b38d411c093a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r" podUID="b16fa692-4b98-42ad-b4db-b38d411c093a" Nov 02 18:15:10.459746 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.457429 5601 scope.go:122] "RemoveContainer" containerID="dd3d38e37961d91273c8ca7451ad8617e8c69bd63daf6788e0b727ee6bbdfe29" Nov 02 18:15:10.503339 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.499427 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" event={"ID":"3e296182-a14a-4ee7-9cd2-d014826b6cd9","Type":"ContainerStarted","Data":"b167c10fdefbb5a742e1d4120dd9b5ce1ec74e490729512a4aead4327e4715be"} Nov 02 18:15:10.503339 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.500011 5601 scope.go:122] "RemoveContainer" containerID="b167c10fdefbb5a742e1d4120dd9b5ce1ec74e490729512a4aead4327e4715be" Nov 02 18:15:10.503339 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:10.500635 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7_odh-ai-gateway-infra(3e296182-a14a-4ee7-9cd2-d014826b6cd9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" podUID="3e296182-a14a-4ee7-9cd2-d014826b6cd9" Nov 02 18:15:10.515094 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.504281 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-workload-socket\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.515094 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.504356 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-data\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.515094 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.504431 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-credential-socket\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.515094 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.504510 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-workload-certs\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.515094 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.504562 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945a032d-c69f-4d9e-96a5-db1ac483af67-istiod-ca-cert\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.515094 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.504659 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-podinfo\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.515094 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.504725 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjdck\" (UniqueName: \"kubernetes.io/projected/945a032d-c69f-4d9e-96a5-db1ac483af67-kube-api-access-zjdck\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.515094 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.504777 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-envoy\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.515094 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.504831 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-token\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.515094 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.507840 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-workload-socket\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.515094 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.509116 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-credential-socket\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.515094 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.509692 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-workload-certs\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.515094 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.510781 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945a032d-c69f-4d9e-96a5-db1ac483af67-istiod-ca-cert\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.515094 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.511808 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-envoy\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.515094 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.513116 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-data\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.515094 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.514921 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-podinfo\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.524301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.522981 5601 generic.go:356] "Generic (PLEG): container finished" podID="62588dd0-b7ee-440e-ab7e-419d7429b2d9" containerID="baada8f707db02645d0e8d44b1acae6d6e42f677bfb65be2553bb80c28fa5800" exitCode=60 Nov 02 18:15:10.524301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.523102 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb" event={"ID":"62588dd0-b7ee-440e-ab7e-419d7429b2d9","Type":"ContainerDied","Data":"baada8f707db02645d0e8d44b1acae6d6e42f677bfb65be2553bb80c28fa5800"} Nov 02 18:15:10.524301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.523681 5601 scope.go:122] "RemoveContainer" containerID="baada8f707db02645d0e8d44b1acae6d6e42f677bfb65be2553bb80c28fa5800" Nov 02 18:15:10.524301 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:10.524200 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb_odh-ai-gateway-infra(62588dd0-b7ee-440e-ab7e-419d7429b2d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb" podUID="62588dd0-b7ee-440e-ab7e-419d7429b2d9" Nov 02 18:15:10.529164 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.529105 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjdck\" (UniqueName: \"kubernetes.io/projected/945a032d-c69f-4d9e-96a5-db1ac483af67-kube-api-access-zjdck\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.532157 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.530844 5601 generic.go:356] "Generic (PLEG): container finished" podID="871bed77-cbe7-4001-a4f8-a6cedca05cef" containerID="a4ef6f0af26e8d9748878198caff03b60b635482bf674d43e0dbada6a528694e" exitCode=60 Nov 02 18:15:10.532157 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.530977 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" event={"ID":"871bed77-cbe7-4001-a4f8-a6cedca05cef","Type":"ContainerDied","Data":"a4ef6f0af26e8d9748878198caff03b60b635482bf674d43e0dbada6a528694e"} Nov 02 18:15:10.540652 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.532584 5601 scope.go:122] "RemoveContainer" containerID="a4ef6f0af26e8d9748878198caff03b60b635482bf674d43e0dbada6a528694e" Nov 02 18:15:10.540652 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:10.533139 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps_odh-ai-gateway-infra(871bed77-cbe7-4001-a4f8-a6cedca05cef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" podUID="871bed77-cbe7-4001-a4f8-a6cedca05cef" Nov 02 18:15:10.579475 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.578677 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-token\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.611920 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.611884 5601 scope.go:122] "RemoveContainer" containerID="fc86dc04699e381711cfd5ece21425d5ecd2f0d948fa04b0ec64a5bf3732c67b" Nov 02 18:15:10.720072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.719950 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:10.731117 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.725344 5601 scope.go:122] "RemoveContainer" containerID="7c1541fdc8df4beeffeddd3b9411e29a9d0217b02e6efaf37a0a9e933a5b67bd" Nov 02 18:15:10.911591 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.911192 5601 scope.go:122] "RemoveContainer" containerID="e036b37a7eb134cc58fd2439dd3c1376d00587ae485f570a273b39d4dbe0bc69" Nov 02 18:15:10.994785 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:10.991810 5601 scope.go:122] "RemoveContainer" containerID="941456ebe919c3ca96ecd4cfec08ee695bd195e45d582098e84ba3e7ee03abf1" Nov 02 18:15:11.017854 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.017828 5601 scope.go:122] "RemoveContainer" containerID="a78f6dab9abdfd2cd41b6ef29f40eb9b0ae45053693fbaf3d7b0ff40afcf53ab" Nov 02 18:15:11.083974 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.083540 5601 scope.go:122] "RemoveContainer" containerID="ec4033eecceea7655a82e4ff13126484df0a56fae0a494d3b75bb6496a8644f7" Nov 02 18:15:11.332380 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:15:11.311823 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945a032d_c69f_4d9e_96a5_db1ac483af67.slice/crio-509a31e45bc5d0fad808fbd674059c9f5272a3c559f880bc93a5d462e93eb0a9 WatchSource:0}: Error finding container 509a31e45bc5d0fad808fbd674059c9f5272a3c559f880bc93a5d462e93eb0a9: Status 404 returned error can't find the container with id 509a31e45bc5d0fad808fbd674059c9f5272a3c559f880bc93a5d462e93eb0a9 Nov 02 18:15:11.343345 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.343293 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz"] Nov 02 18:15:11.404905 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.404845 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:15:11.405332 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.405295 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:15:11.405573 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.405548 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:15:11.447530 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.447483 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a525fd4-e6e8-4ffa-a902-b89637d8d993" path="/var/lib/kubelet/pods/1a525fd4-e6e8-4ffa-a902-b89637d8d993/volumes" Nov 02 18:15:11.448673 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.448643 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46c9a1a3-ff28-4d6c-9d85-faa3be401911" path="/var/lib/kubelet/pods/46c9a1a3-ff28-4d6c-9d85-faa3be401911/volumes" Nov 02 18:15:11.580223 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.580180 5601 scope.go:122] "RemoveContainer" containerID="a4ef6f0af26e8d9748878198caff03b60b635482bf674d43e0dbada6a528694e" Nov 02 18:15:11.580861 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:11.580823 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps_odh-ai-gateway-infra(871bed77-cbe7-4001-a4f8-a6cedca05cef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" podUID="871bed77-cbe7-4001-a4f8-a6cedca05cef" Nov 02 18:15:11.584997 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.584975 5601 scope.go:122] "RemoveContainer" containerID="0649a8c60b016d81ac348477987b182bc6ee6d2e8aaf30f97d41c2f30e5f2882" Nov 02 18:15:11.585338 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:11.585316 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt_odh-ai-gateway-infra(154aa781-e85c-4aad-8fc1-e022e9704801)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt" podUID="154aa781-e85c-4aad-8fc1-e022e9704801" Nov 02 18:15:11.588695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.587190 5601 generic.go:356] "Generic (PLEG): container finished" podID="bd2ceb92-645d-4255-89a8-c0211bc08025" containerID="a70ab00820a19e33994b167d3b6ed533c22b73a5d75eb371e728670d1df11898" exitCode=60 Nov 02 18:15:11.588695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.587250 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" event={"ID":"bd2ceb92-645d-4255-89a8-c0211bc08025","Type":"ContainerDied","Data":"a70ab00820a19e33994b167d3b6ed533c22b73a5d75eb371e728670d1df11898"} Nov 02 18:15:11.588695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.587292 5601 scope.go:122] "RemoveContainer" containerID="a617cfe2c89977a22fb813aee0c665171e4bc6b3391c03ccc28ec0eac6a565c1" Nov 02 18:15:11.588695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.587809 5601 scope.go:122] "RemoveContainer" containerID="a70ab00820a19e33994b167d3b6ed533c22b73a5d75eb371e728670d1df11898" Nov 02 18:15:11.588695 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:11.588266 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj_odh-ai-gateway-infra(bd2ceb92-645d-4255-89a8-c0211bc08025)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" podUID="bd2ceb92-645d-4255-89a8-c0211bc08025" Nov 02 18:15:11.595666 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.594655 5601 scope.go:122] "RemoveContainer" containerID="419216f1f5f94e112f2438bf352d996c57e38f8a03253b24eddcf2fc7eb28b4c" Nov 02 18:15:11.595981 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:11.595874 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s_odh-ai-gateway-infra(aa194666-bb2c-4610-a051-7fd9caef77d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s" podUID="aa194666-bb2c-4610-a051-7fd9caef77d3" Nov 02 18:15:11.603354 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.601892 5601 generic.go:356] "Generic (PLEG): container finished" podID="c4e45226-6362-4b36-b890-11b6804c8d65" containerID="84fb7cb490d4fcef92a8c8d6a4fb3eae8723388817632b899c41482b7ab2b737" exitCode=60 Nov 02 18:15:11.604072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.603976 5601 scope.go:122] "RemoveContainer" containerID="84fb7cb490d4fcef92a8c8d6a4fb3eae8723388817632b899c41482b7ab2b737" Nov 02 18:15:11.609661 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:11.606503 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx_odh-ai-gateway-infra(c4e45226-6362-4b36-b890-11b6804c8d65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" podUID="c4e45226-6362-4b36-b890-11b6804c8d65" Nov 02 18:15:11.609661 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.601988 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" event={"ID":"c4e45226-6362-4b36-b890-11b6804c8d65","Type":"ContainerDied","Data":"84fb7cb490d4fcef92a8c8d6a4fb3eae8723388817632b899c41482b7ab2b737"} Nov 02 18:15:11.609661 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.608961 5601 scope.go:122] "RemoveContainer" containerID="58726700aa56b2ebb545209bb5ac7a1e0795adc04fc61357d8a97c853ae133fa" Nov 02 18:15:11.609661 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:11.609469 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r_odh-ai-gateway-infra(b16fa692-4b98-42ad-b4db-b38d411c093a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r" podUID="b16fa692-4b98-42ad-b4db-b38d411c093a" Nov 02 18:15:11.614312 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.614277 5601 generic.go:356] "Generic (PLEG): container finished" podID="3e296182-a14a-4ee7-9cd2-d014826b6cd9" containerID="b167c10fdefbb5a742e1d4120dd9b5ce1ec74e490729512a4aead4327e4715be" exitCode=60 Nov 02 18:15:11.614514 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.614365 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" event={"ID":"3e296182-a14a-4ee7-9cd2-d014826b6cd9","Type":"ContainerDied","Data":"b167c10fdefbb5a742e1d4120dd9b5ce1ec74e490729512a4aead4327e4715be"} Nov 02 18:15:11.614946 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.614925 5601 scope.go:122] "RemoveContainer" containerID="b167c10fdefbb5a742e1d4120dd9b5ce1ec74e490729512a4aead4327e4715be" Nov 02 18:15:11.615541 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:11.615508 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7_odh-ai-gateway-infra(3e296182-a14a-4ee7-9cd2-d014826b6cd9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" podUID="3e296182-a14a-4ee7-9cd2-d014826b6cd9" Nov 02 18:15:11.631826 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.631292 5601 scope.go:122] "RemoveContainer" containerID="baada8f707db02645d0e8d44b1acae6d6e42f677bfb65be2553bb80c28fa5800" Nov 02 18:15:11.631826 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:11.631701 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb_odh-ai-gateway-infra(62588dd0-b7ee-440e-ab7e-419d7429b2d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb" podUID="62588dd0-b7ee-440e-ab7e-419d7429b2d9" Nov 02 18:15:11.654651 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.643424 5601 scope.go:122] "RemoveContainer" containerID="91d1550e90725e58625868b61b5917a8d3278b510611ad58bf1390a88918704d" Nov 02 18:15:11.655525 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:11.655489 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588_odh-ai-gateway-infra(23af2a44-3074-4dce-a5c6-9c18b4c87d35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588" podUID="23af2a44-3074-4dce-a5c6-9c18b4c87d35" Nov 02 18:15:11.676935 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.676892 5601 scope.go:122] "RemoveContainer" containerID="32529cc757a1be93c0565d2e498efc33d9286fbdefbc09f77219185bba8e8d24" Nov 02 18:15:11.684530 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.684462 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" event={"ID":"945a032d-c69f-4d9e-96a5-db1ac483af67","Type":"ContainerStarted","Data":"509a31e45bc5d0fad808fbd674059c9f5272a3c559f880bc93a5d462e93eb0a9"} Nov 02 18:15:11.697922 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.697199 5601 generic.go:356] "Generic (PLEG): container finished" podID="3611e836-f87d-448f-8f33-5f58724e18cf" containerID="ec824a2722e7687d4acf37ef7f3ff0eaf7dea94d18a7e15d6f33e7a68fa734e7" exitCode=60 Nov 02 18:15:11.697922 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.697303 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg" event={"ID":"3611e836-f87d-448f-8f33-5f58724e18cf","Type":"ContainerDied","Data":"ec824a2722e7687d4acf37ef7f3ff0eaf7dea94d18a7e15d6f33e7a68fa734e7"} Nov 02 18:15:11.697922 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.697891 5601 scope.go:122] "RemoveContainer" containerID="ec824a2722e7687d4acf37ef7f3ff0eaf7dea94d18a7e15d6f33e7a68fa734e7" Nov 02 18:15:11.698380 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:11.698351 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg_odh-ai-gateway-infra(3611e836-f87d-448f-8f33-5f58724e18cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg" podUID="3611e836-f87d-448f-8f33-5f58724e18cf" Nov 02 18:15:11.702309 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.702279 5601 scope.go:122] "RemoveContainer" containerID="e434a331d257538659029e4af02c7db96af646b3599b00fe5754f909ae094b89" Nov 02 18:15:11.703529 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:11.703086 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj_odh-ai-gateway-infra(e2f91fe1-6e31-4c8c-a24e-2d473a179960)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" podUID="e2f91fe1-6e31-4c8c-a24e-2d473a179960" Nov 02 18:15:11.711237 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.711208 5601 scope.go:122] "RemoveContainer" containerID="57632db5054ee6c4f08dbcf0f40865c32a03d531f05f7ae57bfc618c3aa7cb86" Nov 02 18:15:11.721378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.721350 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:11.730205 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.730180 5601 scope.go:122] "RemoveContainer" containerID="4a028e4e6d138c9515be4f2af10eb0923bdca21d5ba79176b82fcaa283827514" Nov 02 18:15:11.820572 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:11.820478 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" podStartSLOduration=3.820457185 podStartE2EDuration="3.820457185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:11.817286101 +0000 UTC m=+2476.508340136" watchObservedRunningTime="2026-11-02 18:15:11.820457185 +0000 UTC m=+2476.511511198" Nov 02 18:15:12.717945 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:12.717916 5601 scope.go:122] "RemoveContainer" containerID="84fb7cb490d4fcef92a8c8d6a4fb3eae8723388817632b899c41482b7ab2b737" Nov 02 18:15:12.718398 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:12.718367 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx_odh-ai-gateway-infra(c4e45226-6362-4b36-b890-11b6804c8d65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" podUID="c4e45226-6362-4b36-b890-11b6804c8d65" Nov 02 18:15:12.719856 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:12.719832 5601 scope.go:122] "RemoveContainer" containerID="b167c10fdefbb5a742e1d4120dd9b5ce1ec74e490729512a4aead4327e4715be" Nov 02 18:15:12.720234 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:12.720204 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7_odh-ai-gateway-infra(3e296182-a14a-4ee7-9cd2-d014826b6cd9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" podUID="3e296182-a14a-4ee7-9cd2-d014826b6cd9" Nov 02 18:15:12.721394 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:12.721369 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" event={"ID":"945a032d-c69f-4d9e-96a5-db1ac483af67","Type":"ContainerStarted","Data":"4f94841406dd48759f969b3f451b0570036eb289bfe818515e0b5e26a9451b3d"} Nov 02 18:15:12.725743 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:12.725721 5601 scope.go:122] "RemoveContainer" containerID="a70ab00820a19e33994b167d3b6ed533c22b73a5d75eb371e728670d1df11898" Nov 02 18:15:12.726033 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:12.726010 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj_odh-ai-gateway-infra(bd2ceb92-645d-4255-89a8-c0211bc08025)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" podUID="bd2ceb92-645d-4255-89a8-c0211bc08025" Nov 02 18:15:12.728831 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:12.728811 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:13.076587 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:13.076543 5601 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17487351_07f6_4585_aa0c_80766361b815.slice/crio-conmon-e9c373df4090677e381f08144a011f7b05d4224e756b54d22fa52275cae1f35b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17487351_07f6_4585_aa0c_80766361b815.slice/crio-e9c373df4090677e381f08144a011f7b05d4224e756b54d22fa52275cae1f35b.scope\": RecentStats: unable to find data in memory cache]" Nov 02 18:15:13.319738 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.319700 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:15:13.399825 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.399727 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17487351-07f6-4585-aa0c-80766361b815-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17487351-07f6-4585-aa0c-80766361b815-istiod-ca-cert\") pod \"17487351-07f6-4585-aa0c-80766361b815\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " Nov 02 18:15:13.400008 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.399867 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-istio-data\") pod \"17487351-07f6-4585-aa0c-80766361b815\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " Nov 02 18:15:13.400008 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.399976 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-workload-certs\") pod \"17487351-07f6-4585-aa0c-80766361b815\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " Nov 02 18:15:13.400148 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.400034 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17487351-07f6-4585-aa0c-80766361b815-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17487351-07f6-4585-aa0c-80766361b815-istio-podinfo\") pod \"17487351-07f6-4585-aa0c-80766361b815\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " Nov 02 18:15:13.400148 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.400067 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17487351-07f6-4585-aa0c-80766361b815-istio-token\" (UniqueName: \"kubernetes.io/projected/17487351-07f6-4585-aa0c-80766361b815-istio-token\") pod \"17487351-07f6-4585-aa0c-80766361b815\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " Nov 02 18:15:13.400148 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.400107 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-workload-socket\") pod \"17487351-07f6-4585-aa0c-80766361b815\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " Nov 02 18:15:13.400313 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.400276 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-istio-data" pod "17487351-07f6-4585-aa0c-80766361b815" (UID: "17487351-07f6-4585-aa0c-80766361b815"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:13.400364 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.400300 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17487351-07f6-4585-aa0c-80766361b815-istiod-ca-cert" pod "17487351-07f6-4585-aa0c-80766361b815" (UID: "17487351-07f6-4585-aa0c-80766361b815"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:13.400364 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.400339 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17487351-07f6-4585-aa0c-80766361b815-kube-api-access-qhdnf\" (UniqueName: \"kubernetes.io/projected/17487351-07f6-4585-aa0c-80766361b815-kube-api-access-qhdnf\") pod \"17487351-07f6-4585-aa0c-80766361b815\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " Nov 02 18:15:13.400468 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.400444 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-credential-socket\") pod \"17487351-07f6-4585-aa0c-80766361b815\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " Nov 02 18:15:13.400525 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.400480 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-istio-envoy\") pod \"17487351-07f6-4585-aa0c-80766361b815\" (UID: \"17487351-07f6-4585-aa0c-80766361b815\") " Nov 02 18:15:13.401304 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.401268 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-credential-socket" pod "17487351-07f6-4585-aa0c-80766361b815" (UID: "17487351-07f6-4585-aa0c-80766361b815"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:13.401514 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.401493 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17487351-07f6-4585-aa0c-80766361b815-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:13.401605 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.401521 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:13.401605 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.401543 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:13.401732 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.401634 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-workload-socket" pod "17487351-07f6-4585-aa0c-80766361b815" (UID: "17487351-07f6-4585-aa0c-80766361b815"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:13.401980 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.401957 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-workload-certs" pod "17487351-07f6-4585-aa0c-80766361b815" (UID: "17487351-07f6-4585-aa0c-80766361b815"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:13.403392 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.403364 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17487351-07f6-4585-aa0c-80766361b815-istio-token" pod "17487351-07f6-4585-aa0c-80766361b815" (UID: "17487351-07f6-4585-aa0c-80766361b815"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:13.404749 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.404589 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17487351-07f6-4585-aa0c-80766361b815-istio-podinfo" pod "17487351-07f6-4585-aa0c-80766361b815" (UID: "17487351-07f6-4585-aa0c-80766361b815"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:15:13.404749 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.404700 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17487351-07f6-4585-aa0c-80766361b815-kube-api-access-qhdnf" pod "17487351-07f6-4585-aa0c-80766361b815" (UID: "17487351-07f6-4585-aa0c-80766361b815"). InnerVolumeSpecName "kube-api-access-qhdnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:13.410551 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.410519 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-istio-envoy" pod "17487351-07f6-4585-aa0c-80766361b815" (UID: "17487351-07f6-4585-aa0c-80766361b815"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:13.504237 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.504187 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:13.504237 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.504237 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17487351-07f6-4585-aa0c-80766361b815-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:13.504473 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.504253 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17487351-07f6-4585-aa0c-80766361b815-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:13.504473 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.504269 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:13.504473 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.504286 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhdnf\" (UniqueName: \"kubernetes.io/projected/17487351-07f6-4585-aa0c-80766361b815-kube-api-access-qhdnf\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:13.504473 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.504301 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17487351-07f6-4585-aa0c-80766361b815-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:13.743972 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.743932 5601 generic.go:356] "Generic (PLEG): container finished" podID="17487351-07f6-4585-aa0c-80766361b815" containerID="e9c373df4090677e381f08144a011f7b05d4224e756b54d22fa52275cae1f35b" exitCode=0 Nov 02 18:15:13.744479 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.744018 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" Nov 02 18:15:13.744479 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.744028 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" event={"ID":"17487351-07f6-4585-aa0c-80766361b815","Type":"ContainerDied","Data":"e9c373df4090677e381f08144a011f7b05d4224e756b54d22fa52275cae1f35b"} Nov 02 18:15:13.744479 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.744096 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml" event={"ID":"17487351-07f6-4585-aa0c-80766361b815","Type":"ContainerDied","Data":"9a073a25fbbeb9fd2d8995a30b3a275241b3a71fa981df55cd440eaf580c1ba0"} Nov 02 18:15:13.744479 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.744127 5601 scope.go:122] "RemoveContainer" containerID="e9c373df4090677e381f08144a011f7b05d4224e756b54d22fa52275cae1f35b" Nov 02 18:15:13.745521 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.744861 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:13.746429 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.746408 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:15:13.758087 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.758060 5601 scope.go:122] "RemoveContainer" containerID="e9c373df4090677e381f08144a011f7b05d4224e756b54d22fa52275cae1f35b" Nov 02 18:15:13.758513 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:13.758485 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c373df4090677e381f08144a011f7b05d4224e756b54d22fa52275cae1f35b\": container with ID starting with e9c373df4090677e381f08144a011f7b05d4224e756b54d22fa52275cae1f35b not found: ID does not exist" containerID="e9c373df4090677e381f08144a011f7b05d4224e756b54d22fa52275cae1f35b" Nov 02 18:15:13.758611 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.758526 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c373df4090677e381f08144a011f7b05d4224e756b54d22fa52275cae1f35b"} err="failed to get container status \"e9c373df4090677e381f08144a011f7b05d4224e756b54d22fa52275cae1f35b\": rpc error: code = NotFound desc = could not find container \"e9c373df4090677e381f08144a011f7b05d4224e756b54d22fa52275cae1f35b\": container with ID starting with e9c373df4090677e381f08144a011f7b05d4224e756b54d22fa52275cae1f35b not found: ID does not exist" Nov 02 18:15:13.804396 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.804350 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml"] Nov 02 18:15:13.834603 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:13.834547 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-gglml"] Nov 02 18:15:14.139890 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.139551 5601 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 18:15:14.139890 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.139611 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" podUID="b7d8a50a-572b-48b7-8c69-93d75c6c78eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 18:15:14.392671 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.392643 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:15:14.442502 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.442458 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-token\" (UniqueName: \"kubernetes.io/projected/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-token\") pod \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " Nov 02 18:15:14.442731 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.442548 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-kube-api-access-z8fbs\" (UniqueName: \"kubernetes.io/projected/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-kube-api-access-z8fbs\") pod \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " Nov 02 18:15:14.442806 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.442750 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-data\") pod \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " Nov 02 18:15:14.442864 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.442815 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istiod-ca-cert\") pod \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " Nov 02 18:15:14.442939 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.442919 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-workload-certs\") pod \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " Nov 02 18:15:14.442994 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.442975 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-podinfo\") pod \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " Nov 02 18:15:14.443047 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.443009 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-envoy\") pod \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " Nov 02 18:15:14.443216 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.443191 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-credential-socket\") pod \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " Nov 02 18:15:14.443288 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.443249 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-workload-socket\") pod \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\" (UID: \"b7d8a50a-572b-48b7-8c69-93d75c6c78eb\") " Nov 02 18:15:14.444863 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.444828 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-workload-certs" pod "b7d8a50a-572b-48b7-8c69-93d75c6c78eb" (UID: "b7d8a50a-572b-48b7-8c69-93d75c6c78eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:14.445018 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.444960 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-workload-socket" pod "b7d8a50a-572b-48b7-8c69-93d75c6c78eb" (UID: "b7d8a50a-572b-48b7-8c69-93d75c6c78eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:14.445113 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.445023 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-credential-socket" pod "b7d8a50a-572b-48b7-8c69-93d75c6c78eb" (UID: "b7d8a50a-572b-48b7-8c69-93d75c6c78eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:14.445113 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.445049 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-data" pod "b7d8a50a-572b-48b7-8c69-93d75c6c78eb" (UID: "b7d8a50a-572b-48b7-8c69-93d75c6c78eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:14.445113 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.445067 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istiod-ca-cert" pod "b7d8a50a-572b-48b7-8c69-93d75c6c78eb" (UID: "b7d8a50a-572b-48b7-8c69-93d75c6c78eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:14.446601 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.446571 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-envoy" pod "b7d8a50a-572b-48b7-8c69-93d75c6c78eb" (UID: "b7d8a50a-572b-48b7-8c69-93d75c6c78eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:14.457805 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.457592 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-kube-api-access-z8fbs" pod "b7d8a50a-572b-48b7-8c69-93d75c6c78eb" (UID: "b7d8a50a-572b-48b7-8c69-93d75c6c78eb"). InnerVolumeSpecName "kube-api-access-z8fbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:14.461127 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.461073 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-token" pod "b7d8a50a-572b-48b7-8c69-93d75c6c78eb" (UID: "b7d8a50a-572b-48b7-8c69-93d75c6c78eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:14.462718 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.462558 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-podinfo" pod "b7d8a50a-572b-48b7-8c69-93d75c6c78eb" (UID: "b7d8a50a-572b-48b7-8c69-93d75c6c78eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:15:14.546136 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.546093 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:14.546136 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.546134 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:14.546358 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.546147 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:14.546358 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.546161 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8fbs\" (UniqueName: \"kubernetes.io/projected/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-kube-api-access-z8fbs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:14.546358 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.546176 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:14.546358 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.546188 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:14.546358 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.546199 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:14.546358 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.546211 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:14.546358 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.546225 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a50a-572b-48b7-8c69-93d75c6c78eb-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:14.757310 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.757263 5601 generic.go:356] "Generic (PLEG): container finished" podID="b7d8a50a-572b-48b7-8c69-93d75c6c78eb" containerID="857937fb5820e416f86f7cc96de9ac48e12f4a5d632bbd6ca10320a6ce9c748c" exitCode=0 Nov 02 18:15:14.757849 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.757337 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" Nov 02 18:15:14.757849 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.757398 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" event={"ID":"b7d8a50a-572b-48b7-8c69-93d75c6c78eb","Type":"ContainerDied","Data":"857937fb5820e416f86f7cc96de9ac48e12f4a5d632bbd6ca10320a6ce9c748c"} Nov 02 18:15:14.757849 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.757451 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj" event={"ID":"b7d8a50a-572b-48b7-8c69-93d75c6c78eb","Type":"ContainerDied","Data":"1fdf6f8a46d74c27e2418a7e9b730e3551e6808be29973b1e17f74936b0cf42f"} Nov 02 18:15:14.757849 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.757488 5601 scope.go:122] "RemoveContainer" containerID="857937fb5820e416f86f7cc96de9ac48e12f4a5d632bbd6ca10320a6ce9c748c" Nov 02 18:15:14.771429 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.771297 5601 scope.go:122] "RemoveContainer" containerID="857937fb5820e416f86f7cc96de9ac48e12f4a5d632bbd6ca10320a6ce9c748c" Nov 02 18:15:14.771875 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:14.771804 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857937fb5820e416f86f7cc96de9ac48e12f4a5d632bbd6ca10320a6ce9c748c\": container with ID starting with 857937fb5820e416f86f7cc96de9ac48e12f4a5d632bbd6ca10320a6ce9c748c not found: ID does not exist" containerID="857937fb5820e416f86f7cc96de9ac48e12f4a5d632bbd6ca10320a6ce9c748c" Nov 02 18:15:14.771875 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.771845 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857937fb5820e416f86f7cc96de9ac48e12f4a5d632bbd6ca10320a6ce9c748c"} err="failed to get container status \"857937fb5820e416f86f7cc96de9ac48e12f4a5d632bbd6ca10320a6ce9c748c\": rpc error: code = NotFound desc = could not find container \"857937fb5820e416f86f7cc96de9ac48e12f4a5d632bbd6ca10320a6ce9c748c\": container with ID starting with 857937fb5820e416f86f7cc96de9ac48e12f4a5d632bbd6ca10320a6ce9c748c not found: ID does not exist" Nov 02 18:15:14.790061 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.790019 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj"] Nov 02 18:15:14.794884 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:14.794848 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-rh5dj"] Nov 02 18:15:15.432382 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.432345 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17487351-07f6-4585-aa0c-80766361b815" path="/var/lib/kubelet/pods/17487351-07f6-4585-aa0c-80766361b815/volumes" Nov 02 18:15:15.433241 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.433218 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d8a50a-572b-48b7-8c69-93d75c6c78eb" path="/var/lib/kubelet/pods/b7d8a50a-572b-48b7-8c69-93d75c6c78eb/volumes" Nov 02 18:15:15.588002 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.587965 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr"] Nov 02 18:15:15.589137 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.589103 5601 cpu_manager.go:414] "containerMap: removing container" podUID="b7d8a50a-572b-48b7-8c69-93d75c6c78eb" containerName="istio-proxy" Nov 02 18:15:15.589137 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.589133 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d8a50a-572b-48b7-8c69-93d75c6c78eb" containerName="istio-proxy" Nov 02 18:15:15.589297 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.589153 5601 cpu_manager.go:414] "containerMap: removing container" podUID="17487351-07f6-4585-aa0c-80766361b815" containerName="istio-proxy" Nov 02 18:15:15.589297 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.589161 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17487351-07f6-4585-aa0c-80766361b815" containerName="istio-proxy" Nov 02 18:15:15.589374 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.589361 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d8a50a-572b-48b7-8c69-93d75c6c78eb" containerName="istio-proxy" Nov 02 18:15:15.589407 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.589379 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="17487351-07f6-4585-aa0c-80766361b815" containerName="istio-proxy" Nov 02 18:15:15.594697 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.594669 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" Nov 02 18:15:15.603635 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.599520 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-312f21\"" Nov 02 18:15:15.603635 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.601006 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-312f21-dockercfg-q7fmb\"" Nov 02 18:15:15.607982 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.606947 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr"] Nov 02 18:15:15.669565 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.669527 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr\" (UID: \"1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" Nov 02 18:15:15.670001 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.669977 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvqz6\" (UniqueName: \"kubernetes.io/projected/1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5-kube-api-access-pvqz6\") pod \"payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr\" (UID: \"1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" Nov 02 18:15:15.844688 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.843814 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr\" (UID: \"1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" Nov 02 18:15:15.844688 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.843883 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr\" (UID: \"1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" Nov 02 18:15:15.844688 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.844123 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvqz6\" (UniqueName: \"kubernetes.io/projected/1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5-kube-api-access-pvqz6\") pod \"payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr\" (UID: \"1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" Nov 02 18:15:15.899643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.894320 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvqz6\" (UniqueName: \"kubernetes.io/projected/1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5-kube-api-access-pvqz6\") pod \"payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr\" (UID: \"1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" Nov 02 18:15:15.919506 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:15.919150 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" Nov 02 18:15:16.326336 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:15:16.326302 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3c2c03_7ac1_4ca4_9c79_3f0bdf8bc5e5.slice/crio-df3ff91b18e6bb47771f6859e1fa73e436da39804dd1f5f571196bf64a40c8cb WatchSource:0}: Error finding container df3ff91b18e6bb47771f6859e1fa73e436da39804dd1f5f571196bf64a40c8cb: Status 404 returned error can't find the container with id df3ff91b18e6bb47771f6859e1fa73e436da39804dd1f5f571196bf64a40c8cb Nov 02 18:15:16.326666 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:16.326637 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr"] Nov 02 18:15:16.869377 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:16.869335 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" event={"ID":"1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5","Type":"ContainerStarted","Data":"ca796f7acc8a4f956cc40f86d3bcbf388a1b9ec2f56c6a2d5cee601c41f64bd8"} Nov 02 18:15:16.869909 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:16.869407 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" event={"ID":"1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5","Type":"ContainerStarted","Data":"df3ff91b18e6bb47771f6859e1fa73e436da39804dd1f5f571196bf64a40c8cb"} Nov 02 18:15:16.869909 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:16.869439 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" Nov 02 18:15:16.892075 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:16.891959 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" podStartSLOduration=1.8919393260000001 podStartE2EDuration="1.891939326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:16.887911154 +0000 UTC m=+2481.578965169" watchObservedRunningTime="2026-11-02 18:15:16.891939326 +0000 UTC m=+2481.582993347" Nov 02 18:15:17.419524 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:17.419489 5601 scope.go:122] "RemoveContainer" containerID="0aa40b01a447acb7b0cb45f8a6ee09e931fdbc447890430266c3cb0f6f447096" Nov 02 18:15:17.818887 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:17.818798 5601 patch_prober.go:29] interesting pod/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 02 18:15:17.818887 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:17.818856 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" podUID="84ed3fb1-d6b4-469c-b7a3-8af367d47208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 02 18:15:17.880167 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:17.880131 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" event={"ID":"18f2b645-9cdf-4b3e-8b54-8e31706d33a3","Type":"ContainerStarted","Data":"0fef44be51f0161132fcbe4977f5dd427c17d7a3e7c836744a84208df5238e00"} Nov 02 18:15:17.882157 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:17.882134 5601 generic.go:356] "Generic (PLEG): container finished" podID="84ed3fb1-d6b4-469c-b7a3-8af367d47208" containerID="75ce8e3cf73639828799ca59a1c1271b21f787c6c1d6803312638ee5ef5db6ff" exitCode=0 Nov 02 18:15:17.882293 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:17.882206 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" event={"ID":"84ed3fb1-d6b4-469c-b7a3-8af367d47208","Type":"ContainerDied","Data":"75ce8e3cf73639828799ca59a1c1271b21f787c6c1d6803312638ee5ef5db6ff"} Nov 02 18:15:17.882774 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:17.882759 5601 scope.go:122] "RemoveContainer" containerID="75ce8e3cf73639828799ca59a1c1271b21f787c6c1d6803312638ee5ef5db6ff" Nov 02 18:15:17.884587 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:17.884559 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:15:17.884692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:17.884639 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:15:17.884692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:17.884674 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:15:18.418733 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:18.418699 5601 scope.go:122] "RemoveContainer" containerID="baada8f707db02645d0e8d44b1acae6d6e42f677bfb65be2553bb80c28fa5800" Nov 02 18:15:18.893337 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:18.893293 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" event={"ID":"84ed3fb1-d6b4-469c-b7a3-8af367d47208","Type":"ContainerStarted","Data":"fb76f1efc01f81666475fa42d6dac18bffc3ce59e041c6b4bc71f4f844a208fa"} Nov 02 18:15:18.895790 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:18.895757 5601 generic.go:356] "Generic (PLEG): container finished" podID="18f2b645-9cdf-4b3e-8b54-8e31706d33a3" containerID="0fef44be51f0161132fcbe4977f5dd427c17d7a3e7c836744a84208df5238e00" exitCode=60 Nov 02 18:15:18.895908 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:18.895828 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" event={"ID":"18f2b645-9cdf-4b3e-8b54-8e31706d33a3","Type":"ContainerDied","Data":"0fef44be51f0161132fcbe4977f5dd427c17d7a3e7c836744a84208df5238e00"} Nov 02 18:15:18.895908 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:18.895888 5601 scope.go:122] "RemoveContainer" containerID="0aa40b01a447acb7b0cb45f8a6ee09e931fdbc447890430266c3cb0f6f447096" Nov 02 18:15:18.896357 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:18.896340 5601 scope.go:122] "RemoveContainer" containerID="0fef44be51f0161132fcbe4977f5dd427c17d7a3e7c836744a84208df5238e00" Nov 02 18:15:18.897178 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:18.897140 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf_odh-ai-gateway-infra(18f2b645-9cdf-4b3e-8b54-8e31706d33a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" podUID="18f2b645-9cdf-4b3e-8b54-8e31706d33a3" Nov 02 18:15:18.909430 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:18.909320 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf"] Nov 02 18:15:19.201671 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:19.191610 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:15:19.201671 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:19.191768 5601 patch_prober.go:29] interesting pod/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 02 18:15:19.201671 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:19.191817 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" podUID="84ed3fb1-d6b4-469c-b7a3-8af367d47208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 02 18:15:19.419075 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:19.419048 5601 scope.go:122] "RemoveContainer" containerID="d93a689c9cfa4fe350d2c16a976c8835fdb77ccd8942a8770685635bb2de2939" Nov 02 18:15:19.419248 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:19.419200 5601 scope.go:122] "RemoveContainer" containerID="91d1550e90725e58625868b61b5917a8d3278b510611ad58bf1390a88918704d" Nov 02 18:15:19.907469 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:19.907434 5601 generic.go:356] "Generic (PLEG): container finished" podID="62588dd0-b7ee-440e-ab7e-419d7429b2d9" containerID="32cf4eeb9d02c40c1f86451ea042cab8c0e2ab2ba3b1f4bd9639c769d26ebeb3" exitCode=60 Nov 02 18:15:19.907888 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:19.907606 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb" event={"ID":"62588dd0-b7ee-440e-ab7e-419d7429b2d9","Type":"ContainerDied","Data":"32cf4eeb9d02c40c1f86451ea042cab8c0e2ab2ba3b1f4bd9639c769d26ebeb3"} Nov 02 18:15:19.907888 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:19.907691 5601 scope.go:122] "RemoveContainer" containerID="baada8f707db02645d0e8d44b1acae6d6e42f677bfb65be2553bb80c28fa5800" Nov 02 18:15:19.908088 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:19.908073 5601 scope.go:122] "RemoveContainer" containerID="32cf4eeb9d02c40c1f86451ea042cab8c0e2ab2ba3b1f4bd9639c769d26ebeb3" Nov 02 18:15:19.908462 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:19.908444 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb_odh-ai-gateway-infra(62588dd0-b7ee-440e-ab7e-419d7429b2d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb" podUID="62588dd0-b7ee-440e-ab7e-419d7429b2d9" Nov 02 18:15:20.191396 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.190947 5601 patch_prober.go:29] interesting pod/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 02 18:15:20.191396 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.191005 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" podUID="84ed3fb1-d6b4-469c-b7a3-8af367d47208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 02 18:15:20.231550 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.231518 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" Nov 02 18:15:20.420004 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.419967 5601 scope.go:122] "RemoveContainer" containerID="419216f1f5f94e112f2438bf352d996c57e38f8a03253b24eddcf2fc7eb28b4c" Nov 02 18:15:20.420224 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.420179 5601 scope.go:122] "RemoveContainer" containerID="cde0423afcc4c2495be038a4819183dfced23f0e623f175b645d61a806897537" Nov 02 18:15:20.420741 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.420703 5601 scope.go:122] "RemoveContainer" containerID="0649a8c60b016d81ac348477987b182bc6ee6d2e8aaf30f97d41c2f30e5f2882" Nov 02 18:15:20.431349 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.431322 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18f2b645-9cdf-4b3e-8b54-8e31706d33a3-kube-api-access-zzs7t\" (UniqueName: \"kubernetes.io/projected/18f2b645-9cdf-4b3e-8b54-8e31706d33a3-kube-api-access-zzs7t\") pod \"18f2b645-9cdf-4b3e-8b54-8e31706d33a3\" (UID: \"18f2b645-9cdf-4b3e-8b54-8e31706d33a3\") " Nov 02 18:15:20.431834 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.431793 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18f2b645-9cdf-4b3e-8b54-8e31706d33a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18f2b645-9cdf-4b3e-8b54-8e31706d33a3-maas-api-service-ca\") pod \"18f2b645-9cdf-4b3e-8b54-8e31706d33a3\" (UID: \"18f2b645-9cdf-4b3e-8b54-8e31706d33a3\") " Nov 02 18:15:20.432137 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.432112 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18f2b645-9cdf-4b3e-8b54-8e31706d33a3-maas-api-service-ca" pod "18f2b645-9cdf-4b3e-8b54-8e31706d33a3" (UID: "18f2b645-9cdf-4b3e-8b54-8e31706d33a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:20.432989 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.432966 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18f2b645-9cdf-4b3e-8b54-8e31706d33a3-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:20.464690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.455195 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18f2b645-9cdf-4b3e-8b54-8e31706d33a3-kube-api-access-zzs7t" pod "18f2b645-9cdf-4b3e-8b54-8e31706d33a3" (UID: "18f2b645-9cdf-4b3e-8b54-8e31706d33a3"). InnerVolumeSpecName "kube-api-access-zzs7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:20.564643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.564040 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzs7t\" (UniqueName: \"kubernetes.io/projected/18f2b645-9cdf-4b3e-8b54-8e31706d33a3-kube-api-access-zzs7t\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:20.931040 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.928852 5601 generic.go:356] "Generic (PLEG): container finished" podID="23af2a44-3074-4dce-a5c6-9c18b4c87d35" containerID="3dabf492c2a9372c0ee4a9a2055570c5a209ca77fd00e8f19d796609750dec94" exitCode=60 Nov 02 18:15:20.931040 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.928972 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588" event={"ID":"23af2a44-3074-4dce-a5c6-9c18b4c87d35","Type":"ContainerDied","Data":"3dabf492c2a9372c0ee4a9a2055570c5a209ca77fd00e8f19d796609750dec94"} Nov 02 18:15:20.931040 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.929015 5601 scope.go:122] "RemoveContainer" containerID="91d1550e90725e58625868b61b5917a8d3278b510611ad58bf1390a88918704d" Nov 02 18:15:20.931040 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.929424 5601 scope.go:122] "RemoveContainer" containerID="3dabf492c2a9372c0ee4a9a2055570c5a209ca77fd00e8f19d796609750dec94" Nov 02 18:15:20.931040 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:20.929859 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588_odh-ai-gateway-infra(23af2a44-3074-4dce-a5c6-9c18b4c87d35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588" podUID="23af2a44-3074-4dce-a5c6-9c18b4c87d35" Nov 02 18:15:20.933208 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.933181 5601 generic.go:356] "Generic (PLEG): container finished" podID="6612ae11-596f-4056-b6bc-42ed18333072" containerID="05bcaa1af5b24944281ee1372f080d7633b47fa894e62969d9cfe01d29b8b851" exitCode=60 Nov 02 18:15:20.933328 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.933256 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j" event={"ID":"6612ae11-596f-4056-b6bc-42ed18333072","Type":"ContainerDied","Data":"05bcaa1af5b24944281ee1372f080d7633b47fa894e62969d9cfe01d29b8b851"} Nov 02 18:15:20.939993 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.933757 5601 scope.go:122] "RemoveContainer" containerID="05bcaa1af5b24944281ee1372f080d7633b47fa894e62969d9cfe01d29b8b851" Nov 02 18:15:20.939993 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:20.934228 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j_odh-ai-gateway-infra(6612ae11-596f-4056-b6bc-42ed18333072)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j" podUID="6612ae11-596f-4056-b6bc-42ed18333072" Nov 02 18:15:20.956673 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.954705 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb"] Nov 02 18:15:20.957055 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.956672 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" event={"ID":"18f2b645-9cdf-4b3e-8b54-8e31706d33a3","Type":"ContainerDied","Data":"6fe6fe29490542fd05fa822117c5e89807b0bddc312340af46de7717833ade2c"} Nov 02 18:15:20.957055 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.956773 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf" Nov 02 18:15:20.981602 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.981577 5601 scope.go:122] "RemoveContainer" containerID="d93a689c9cfa4fe350d2c16a976c8835fdb77ccd8942a8770685635bb2de2939" Nov 02 18:15:20.998646 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:20.998596 5601 scope.go:122] "RemoveContainer" containerID="0fef44be51f0161132fcbe4977f5dd427c17d7a3e7c836744a84208df5238e00" Nov 02 18:15:21.013225 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.013195 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf"] Nov 02 18:15:21.018018 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.017988 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2961d5-29894055-c44mf"] Nov 02 18:15:21.192644 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.190100 5601 patch_prober.go:29] interesting pod/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 02 18:15:21.192644 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.190166 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" podUID="84ed3fb1-d6b4-469c-b7a3-8af367d47208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 02 18:15:21.289532 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.289499 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb" Nov 02 18:15:21.386908 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.386868 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62588dd0-b7ee-440e-ab7e-419d7429b2d9-kube-api-access-t52l7\" (UniqueName: \"kubernetes.io/projected/62588dd0-b7ee-440e-ab7e-419d7429b2d9-kube-api-access-t52l7\") pod \"62588dd0-b7ee-440e-ab7e-419d7429b2d9\" (UID: \"62588dd0-b7ee-440e-ab7e-419d7429b2d9\") " Nov 02 18:15:21.387124 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.387009 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62588dd0-b7ee-440e-ab7e-419d7429b2d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62588dd0-b7ee-440e-ab7e-419d7429b2d9-maas-api-service-ca\") pod \"62588dd0-b7ee-440e-ab7e-419d7429b2d9\" (UID: \"62588dd0-b7ee-440e-ab7e-419d7429b2d9\") " Nov 02 18:15:21.387753 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.387727 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62588dd0-b7ee-440e-ab7e-419d7429b2d9-maas-api-service-ca" pod "62588dd0-b7ee-440e-ab7e-419d7429b2d9" (UID: "62588dd0-b7ee-440e-ab7e-419d7429b2d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:21.389829 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.389805 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62588dd0-b7ee-440e-ab7e-419d7429b2d9-kube-api-access-t52l7" pod "62588dd0-b7ee-440e-ab7e-419d7429b2d9" (UID: "62588dd0-b7ee-440e-ab7e-419d7429b2d9"). InnerVolumeSpecName "kube-api-access-t52l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:21.419655 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.419604 5601 scope.go:122] "RemoveContainer" containerID="a4ef6f0af26e8d9748878198caff03b60b635482bf674d43e0dbada6a528694e" Nov 02 18:15:21.420068 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.420043 5601 scope.go:122] "RemoveContainer" containerID="ec824a2722e7687d4acf37ef7f3ff0eaf7dea94d18a7e15d6f33e7a68fa734e7" Nov 02 18:15:21.420648 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.420609 5601 scope.go:122] "RemoveContainer" containerID="e434a331d257538659029e4af02c7db96af646b3599b00fe5754f909ae094b89" Nov 02 18:15:21.422779 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.422411 5601 scope.go:122] "RemoveContainer" containerID="84fb7cb490d4fcef92a8c8d6a4fb3eae8723388817632b899c41482b7ab2b737" Nov 02 18:15:21.423015 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.422994 5601 scope.go:122] "RemoveContainer" containerID="58726700aa56b2ebb545209bb5ac7a1e0795adc04fc61357d8a97c853ae133fa" Nov 02 18:15:21.455641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.433662 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18f2b645-9cdf-4b3e-8b54-8e31706d33a3" path="/var/lib/kubelet/pods/18f2b645-9cdf-4b3e-8b54-8e31706d33a3/volumes" Nov 02 18:15:21.491270 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.489896 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t52l7\" (UniqueName: \"kubernetes.io/projected/62588dd0-b7ee-440e-ab7e-419d7429b2d9-kube-api-access-t52l7\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:21.491270 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.489926 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62588dd0-b7ee-440e-ab7e-419d7429b2d9-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:21.971853 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.971816 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588"] Nov 02 18:15:21.980546 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.980518 5601 generic.go:356] "Generic (PLEG): container finished" podID="e6aed869-903a-4e44-aaed-66d8940f4f90" containerID="7d1baeda98d76b305bd8fa34066a748479866510f898237d065892512669db54" exitCode=60 Nov 02 18:15:21.980711 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.980657 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z" event={"ID":"e6aed869-903a-4e44-aaed-66d8940f4f90","Type":"ContainerDied","Data":"7d1baeda98d76b305bd8fa34066a748479866510f898237d065892512669db54"} Nov 02 18:15:21.980760 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.980714 5601 scope.go:122] "RemoveContainer" containerID="cde0423afcc4c2495be038a4819183dfced23f0e623f175b645d61a806897537" Nov 02 18:15:21.981213 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.981190 5601 scope.go:122] "RemoveContainer" containerID="7d1baeda98d76b305bd8fa34066a748479866510f898237d065892512669db54" Nov 02 18:15:21.981686 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:21.981657 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z_odh-ai-gateway-infra(e6aed869-903a-4e44-aaed-66d8940f4f90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z" podUID="e6aed869-903a-4e44-aaed-66d8940f4f90" Nov 02 18:15:21.993261 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.993232 5601 generic.go:356] "Generic (PLEG): container finished" podID="154aa781-e85c-4aad-8fc1-e022e9704801" containerID="1463ef2fe119059aba7e8b3bb7cb8078693bc36d8c69ac00255cc0a966fbba9d" exitCode=60 Nov 02 18:15:21.996696 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.993412 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt" event={"ID":"154aa781-e85c-4aad-8fc1-e022e9704801","Type":"ContainerDied","Data":"1463ef2fe119059aba7e8b3bb7cb8078693bc36d8c69ac00255cc0a966fbba9d"} Nov 02 18:15:21.996696 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:21.996109 5601 scope.go:122] "RemoveContainer" containerID="1463ef2fe119059aba7e8b3bb7cb8078693bc36d8c69ac00255cc0a966fbba9d" Nov 02 18:15:21.996696 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:21.996565 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt_odh-ai-gateway-infra(154aa781-e85c-4aad-8fc1-e022e9704801)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt" podUID="154aa781-e85c-4aad-8fc1-e022e9704801" Nov 02 18:15:22.005497 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.005465 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j"] Nov 02 18:15:22.010526 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.010402 5601 scope.go:122] "RemoveContainer" containerID="0649a8c60b016d81ac348477987b182bc6ee6d2e8aaf30f97d41c2f30e5f2882" Nov 02 18:15:22.011837 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.011575 5601 generic.go:356] "Generic (PLEG): container finished" podID="aa194666-bb2c-4610-a051-7fd9caef77d3" containerID="11209dcee18d14c142716ffbcb3f993d2c5634b68886103fd482a9995b34747d" exitCode=60 Nov 02 18:15:22.011837 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.011677 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s" event={"ID":"aa194666-bb2c-4610-a051-7fd9caef77d3","Type":"ContainerDied","Data":"11209dcee18d14c142716ffbcb3f993d2c5634b68886103fd482a9995b34747d"} Nov 02 18:15:22.012331 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.012170 5601 scope.go:122] "RemoveContainer" containerID="11209dcee18d14c142716ffbcb3f993d2c5634b68886103fd482a9995b34747d" Nov 02 18:15:22.014761 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:22.012676 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s_odh-ai-gateway-infra(aa194666-bb2c-4610-a051-7fd9caef77d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s" podUID="aa194666-bb2c-4610-a051-7fd9caef77d3" Nov 02 18:15:22.015278 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.015254 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb" event={"ID":"62588dd0-b7ee-440e-ab7e-419d7429b2d9","Type":"ContainerDied","Data":"50ff4ad8fd522c0e968a0339ca77a968336c04d0ff3c7dccddcab8039b1d3fac"} Nov 02 18:15:22.015526 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.015464 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb" Nov 02 18:15:22.069859 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.069830 5601 scope.go:122] "RemoveContainer" containerID="419216f1f5f94e112f2438bf352d996c57e38f8a03253b24eddcf2fc7eb28b4c" Nov 02 18:15:22.091452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.090912 5601 scope.go:122] "RemoveContainer" containerID="32cf4eeb9d02c40c1f86451ea042cab8c0e2ab2ba3b1f4bd9639c769d26ebeb3" Nov 02 18:15:22.172221 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.172174 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb"] Nov 02 18:15:22.178353 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.178311 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894055-98tnb"] Nov 02 18:15:22.190359 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.190335 5601 patch_prober.go:29] interesting pod/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 02 18:15:22.190485 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.190383 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" podUID="84ed3fb1-d6b4-469c-b7a3-8af367d47208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 02 18:15:22.305316 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.304968 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588" Nov 02 18:15:22.406467 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.406368 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23af2a44-3074-4dce-a5c6-9c18b4c87d35-kube-api-access-wjgsw\" (UniqueName: \"kubernetes.io/projected/23af2a44-3074-4dce-a5c6-9c18b4c87d35-kube-api-access-wjgsw\") pod \"23af2a44-3074-4dce-a5c6-9c18b4c87d35\" (UID: \"23af2a44-3074-4dce-a5c6-9c18b4c87d35\") " Nov 02 18:15:22.408424 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.408231 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23af2a44-3074-4dce-a5c6-9c18b4c87d35-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23af2a44-3074-4dce-a5c6-9c18b4c87d35-maas-api-service-ca\") pod \"23af2a44-3074-4dce-a5c6-9c18b4c87d35\" (UID: \"23af2a44-3074-4dce-a5c6-9c18b4c87d35\") " Nov 02 18:15:22.408733 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.408708 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23af2a44-3074-4dce-a5c6-9c18b4c87d35-maas-api-service-ca" pod "23af2a44-3074-4dce-a5c6-9c18b4c87d35" (UID: "23af2a44-3074-4dce-a5c6-9c18b4c87d35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:22.408826 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.408794 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23af2a44-3074-4dce-a5c6-9c18b4c87d35-kube-api-access-wjgsw" pod "23af2a44-3074-4dce-a5c6-9c18b4c87d35" (UID: "23af2a44-3074-4dce-a5c6-9c18b4c87d35"). InnerVolumeSpecName "kube-api-access-wjgsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:22.409808 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.409752 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjgsw\" (UniqueName: \"kubernetes.io/projected/23af2a44-3074-4dce-a5c6-9c18b4c87d35-kube-api-access-wjgsw\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:22.409808 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.409782 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23af2a44-3074-4dce-a5c6-9c18b4c87d35-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:22.419060 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.418989 5601 scope.go:122] "RemoveContainer" containerID="b167c10fdefbb5a742e1d4120dd9b5ce1ec74e490729512a4aead4327e4715be" Nov 02 18:15:22.425269 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.424773 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j" Nov 02 18:15:22.510426 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.510386 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6612ae11-596f-4056-b6bc-42ed18333072-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6612ae11-596f-4056-b6bc-42ed18333072-maas-api-service-ca\") pod \"6612ae11-596f-4056-b6bc-42ed18333072\" (UID: \"6612ae11-596f-4056-b6bc-42ed18333072\") " Nov 02 18:15:22.510758 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.510733 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6612ae11-596f-4056-b6bc-42ed18333072-kube-api-access-78898\" (UniqueName: \"kubernetes.io/projected/6612ae11-596f-4056-b6bc-42ed18333072-kube-api-access-78898\") pod \"6612ae11-596f-4056-b6bc-42ed18333072\" (UID: \"6612ae11-596f-4056-b6bc-42ed18333072\") " Nov 02 18:15:22.510868 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.510843 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6612ae11-596f-4056-b6bc-42ed18333072-maas-api-service-ca" pod "6612ae11-596f-4056-b6bc-42ed18333072" (UID: "6612ae11-596f-4056-b6bc-42ed18333072"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:22.511706 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.511683 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6612ae11-596f-4056-b6bc-42ed18333072-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:22.513529 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.513499 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6612ae11-596f-4056-b6bc-42ed18333072-kube-api-access-78898" pod "6612ae11-596f-4056-b6bc-42ed18333072" (UID: "6612ae11-596f-4056-b6bc-42ed18333072"). InnerVolumeSpecName "kube-api-access-78898". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:22.626683 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:22.625576 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78898\" (UniqueName: \"kubernetes.io/projected/6612ae11-596f-4056-b6bc-42ed18333072-kube-api-access-78898\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:23.033678 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.033639 5601 generic.go:356] "Generic (PLEG): container finished" podID="c4e45226-6362-4b36-b890-11b6804c8d65" containerID="6ac2da68f625a06efc6c151d7e2add962d3378328248a9b1ab34565dcf1c2114" exitCode=60 Nov 02 18:15:23.049588 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.033735 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" event={"ID":"c4e45226-6362-4b36-b890-11b6804c8d65","Type":"ContainerDied","Data":"6ac2da68f625a06efc6c151d7e2add962d3378328248a9b1ab34565dcf1c2114"} Nov 02 18:15:23.049588 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.033790 5601 scope.go:122] "RemoveContainer" containerID="84fb7cb490d4fcef92a8c8d6a4fb3eae8723388817632b899c41482b7ab2b737" Nov 02 18:15:23.049588 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.034202 5601 scope.go:122] "RemoveContainer" containerID="6ac2da68f625a06efc6c151d7e2add962d3378328248a9b1ab34565dcf1c2114" Nov 02 18:15:23.049588 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:23.034535 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx_odh-ai-gateway-infra(c4e45226-6362-4b36-b890-11b6804c8d65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" podUID="c4e45226-6362-4b36-b890-11b6804c8d65" Nov 02 18:15:23.049588 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.037778 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z"] Nov 02 18:15:23.088661 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.088599 5601 generic.go:356] "Generic (PLEG): container finished" podID="b16fa692-4b98-42ad-b4db-b38d411c093a" containerID="7e82c16a766f2d4cbc9c540655976f65523cfe703a2cc13784c9a1ea74bd5a26" exitCode=60 Nov 02 18:15:23.088870 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.088839 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r" event={"ID":"b16fa692-4b98-42ad-b4db-b38d411c093a","Type":"ContainerDied","Data":"7e82c16a766f2d4cbc9c540655976f65523cfe703a2cc13784c9a1ea74bd5a26"} Nov 02 18:15:23.089530 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.089503 5601 scope.go:122] "RemoveContainer" containerID="7e82c16a766f2d4cbc9c540655976f65523cfe703a2cc13784c9a1ea74bd5a26" Nov 02 18:15:23.090178 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:23.090146 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r_odh-ai-gateway-infra(b16fa692-4b98-42ad-b4db-b38d411c093a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r" podUID="b16fa692-4b98-42ad-b4db-b38d411c093a" Nov 02 18:15:23.094198 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.094169 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" event={"ID":"3e296182-a14a-4ee7-9cd2-d014826b6cd9","Type":"ContainerStarted","Data":"c31127505e4f0a9d283607620299b193fe588ac8a0d0a76d24847cedb2edffaf"} Nov 02 18:15:23.099395 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.099364 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588" event={"ID":"23af2a44-3074-4dce-a5c6-9c18b4c87d35","Type":"ContainerDied","Data":"d02b2000b0934f024242254fc1afd706b4f20ae7b89c2b1bc021f10a4781708a"} Nov 02 18:15:23.099535 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.099463 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588" Nov 02 18:15:23.105460 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.104663 5601 generic.go:356] "Generic (PLEG): container finished" podID="3611e836-f87d-448f-8f33-5f58724e18cf" containerID="8184486db8cb724f33026c7336b196be283fb4a3f200585a9465e9a00fff7a2d" exitCode=60 Nov 02 18:15:23.105460 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.104772 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg" event={"ID":"3611e836-f87d-448f-8f33-5f58724e18cf","Type":"ContainerDied","Data":"8184486db8cb724f33026c7336b196be283fb4a3f200585a9465e9a00fff7a2d"} Nov 02 18:15:23.105460 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.105264 5601 scope.go:122] "RemoveContainer" containerID="8184486db8cb724f33026c7336b196be283fb4a3f200585a9465e9a00fff7a2d" Nov 02 18:15:23.109553 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.109532 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt"] Nov 02 18:15:23.109940 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:23.109913 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg_odh-ai-gateway-infra(3611e836-f87d-448f-8f33-5f58724e18cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg" podUID="3611e836-f87d-448f-8f33-5f58724e18cf" Nov 02 18:15:23.115362 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.114852 5601 scope.go:122] "RemoveContainer" containerID="58726700aa56b2ebb545209bb5ac7a1e0795adc04fc61357d8a97c853ae133fa" Nov 02 18:15:23.120021 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.119990 5601 generic.go:356] "Generic (PLEG): container finished" podID="e2f91fe1-6e31-4c8c-a24e-2d473a179960" containerID="cc0166be9a68eb70d8468a0d9a04086bc0f0ef3434b946ec20da7f1beb065499" exitCode=60 Nov 02 18:15:23.120218 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.120055 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" event={"ID":"e2f91fe1-6e31-4c8c-a24e-2d473a179960","Type":"ContainerDied","Data":"cc0166be9a68eb70d8468a0d9a04086bc0f0ef3434b946ec20da7f1beb065499"} Nov 02 18:15:23.120795 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.120775 5601 scope.go:122] "RemoveContainer" containerID="cc0166be9a68eb70d8468a0d9a04086bc0f0ef3434b946ec20da7f1beb065499" Nov 02 18:15:23.121225 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:23.121203 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj_odh-ai-gateway-infra(e2f91fe1-6e31-4c8c-a24e-2d473a179960)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" podUID="e2f91fe1-6e31-4c8c-a24e-2d473a179960" Nov 02 18:15:23.122916 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.122897 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j" Nov 02 18:15:23.122916 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.122907 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j" event={"ID":"6612ae11-596f-4056-b6bc-42ed18333072","Type":"ContainerDied","Data":"c93399e5267cf3e3153b482aa16b77740f472b17beb07049ff9accc567245355"} Nov 02 18:15:23.123356 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.123338 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s"] Nov 02 18:15:23.132422 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.131480 5601 generic.go:356] "Generic (PLEG): container finished" podID="871bed77-cbe7-4001-a4f8-a6cedca05cef" containerID="b98049a97fc1752ec230ebe0fee5b4536cfc6a8a991d3713886b70cc4d7562e5" exitCode=60 Nov 02 18:15:23.132422 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.131578 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" event={"ID":"871bed77-cbe7-4001-a4f8-a6cedca05cef","Type":"ContainerDied","Data":"b98049a97fc1752ec230ebe0fee5b4536cfc6a8a991d3713886b70cc4d7562e5"} Nov 02 18:15:23.132422 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.132114 5601 scope.go:122] "RemoveContainer" containerID="b98049a97fc1752ec230ebe0fee5b4536cfc6a8a991d3713886b70cc4d7562e5" Nov 02 18:15:23.133526 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:23.133494 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps_odh-ai-gateway-infra(871bed77-cbe7-4001-a4f8-a6cedca05cef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" podUID="871bed77-cbe7-4001-a4f8-a6cedca05cef" Nov 02 18:15:23.172790 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.172760 5601 scope.go:122] "RemoveContainer" containerID="3dabf492c2a9372c0ee4a9a2055570c5a209ca77fd00e8f19d796609750dec94" Nov 02 18:15:23.180295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.180227 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588"] Nov 02 18:15:23.187452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.187389 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894055-kp588"] Nov 02 18:15:23.190632 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.190581 5601 scope.go:122] "RemoveContainer" containerID="ec824a2722e7687d4acf37ef7f3ff0eaf7dea94d18a7e15d6f33e7a68fa734e7" Nov 02 18:15:23.191751 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.191059 5601 patch_prober.go:29] interesting pod/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 02 18:15:23.191751 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.191113 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" podUID="84ed3fb1-d6b4-469c-b7a3-8af367d47208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 02 18:15:23.220139 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.220110 5601 scope.go:122] "RemoveContainer" containerID="e434a331d257538659029e4af02c7db96af646b3599b00fe5754f909ae094b89" Nov 02 18:15:23.220802 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.220558 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j"] Nov 02 18:15:23.229339 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.228315 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a8e574-29894055-8vw9j"] Nov 02 18:15:23.287179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.287152 5601 scope.go:122] "RemoveContainer" containerID="05bcaa1af5b24944281ee1372f080d7633b47fa894e62969d9cfe01d29b8b851" Nov 02 18:15:23.327318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.326359 5601 scope.go:122] "RemoveContainer" containerID="a4ef6f0af26e8d9748878198caff03b60b635482bf674d43e0dbada6a528694e" Nov 02 18:15:23.419794 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.419768 5601 scope.go:122] "RemoveContainer" containerID="a70ab00820a19e33994b167d3b6ed533c22b73a5d75eb371e728670d1df11898" Nov 02 18:15:23.433454 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.433420 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z" Nov 02 18:15:23.615678 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.615584 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23af2a44-3074-4dce-a5c6-9c18b4c87d35" path="/var/lib/kubelet/pods/23af2a44-3074-4dce-a5c6-9c18b4c87d35/volumes" Nov 02 18:15:23.617083 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.616956 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62588dd0-b7ee-440e-ab7e-419d7429b2d9" path="/var/lib/kubelet/pods/62588dd0-b7ee-440e-ab7e-419d7429b2d9/volumes" Nov 02 18:15:23.619702 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.617609 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6612ae11-596f-4056-b6bc-42ed18333072" path="/var/lib/kubelet/pods/6612ae11-596f-4056-b6bc-42ed18333072/volumes" Nov 02 18:15:23.758729 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.756468 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6aed869-903a-4e44-aaed-66d8940f4f90-kube-api-access-nmk7d\" (UniqueName: \"kubernetes.io/projected/e6aed869-903a-4e44-aaed-66d8940f4f90-kube-api-access-nmk7d\") pod \"e6aed869-903a-4e44-aaed-66d8940f4f90\" (UID: \"e6aed869-903a-4e44-aaed-66d8940f4f90\") " Nov 02 18:15:23.758729 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.756670 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6aed869-903a-4e44-aaed-66d8940f4f90-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6aed869-903a-4e44-aaed-66d8940f4f90-maas-api-service-ca\") pod \"e6aed869-903a-4e44-aaed-66d8940f4f90\" (UID: \"e6aed869-903a-4e44-aaed-66d8940f4f90\") " Nov 02 18:15:23.759341 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.759293 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6aed869-903a-4e44-aaed-66d8940f4f90-maas-api-service-ca" pod "e6aed869-903a-4e44-aaed-66d8940f4f90" (UID: "e6aed869-903a-4e44-aaed-66d8940f4f90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:23.776717 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.766044 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6aed869-903a-4e44-aaed-66d8940f4f90-kube-api-access-nmk7d" pod "e6aed869-903a-4e44-aaed-66d8940f4f90" (UID: "e6aed869-903a-4e44-aaed-66d8940f4f90"). InnerVolumeSpecName "kube-api-access-nmk7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:23.784075 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.784044 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt" Nov 02 18:15:23.788608 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.788576 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s" Nov 02 18:15:23.859157 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.859118 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6aed869-903a-4e44-aaed-66d8940f4f90-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:23.859157 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.859160 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmk7d\" (UniqueName: \"kubernetes.io/projected/e6aed869-903a-4e44-aaed-66d8940f4f90-kube-api-access-nmk7d\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:23.964904 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.964573 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa194666-bb2c-4610-a051-7fd9caef77d3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa194666-bb2c-4610-a051-7fd9caef77d3-maas-api-service-ca\") pod \"aa194666-bb2c-4610-a051-7fd9caef77d3\" (UID: \"aa194666-bb2c-4610-a051-7fd9caef77d3\") " Nov 02 18:15:23.964904 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.964661 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa194666-bb2c-4610-a051-7fd9caef77d3-kube-api-access-g8pxg\" (UniqueName: \"kubernetes.io/projected/aa194666-bb2c-4610-a051-7fd9caef77d3-kube-api-access-g8pxg\") pod \"aa194666-bb2c-4610-a051-7fd9caef77d3\" (UID: \"aa194666-bb2c-4610-a051-7fd9caef77d3\") " Nov 02 18:15:23.965252 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.965210 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa194666-bb2c-4610-a051-7fd9caef77d3-maas-api-service-ca" pod "aa194666-bb2c-4610-a051-7fd9caef77d3" (UID: "aa194666-bb2c-4610-a051-7fd9caef77d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:23.965799 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.965775 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/154aa781-e85c-4aad-8fc1-e022e9704801-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/154aa781-e85c-4aad-8fc1-e022e9704801-maas-api-service-ca\") pod \"154aa781-e85c-4aad-8fc1-e022e9704801\" (UID: \"154aa781-e85c-4aad-8fc1-e022e9704801\") " Nov 02 18:15:23.966042 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.966023 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154aa781-e85c-4aad-8fc1-e022e9704801-kube-api-access-q795d\" (UniqueName: \"kubernetes.io/projected/154aa781-e85c-4aad-8fc1-e022e9704801-kube-api-access-q795d\") pod \"154aa781-e85c-4aad-8fc1-e022e9704801\" (UID: \"154aa781-e85c-4aad-8fc1-e022e9704801\") " Nov 02 18:15:23.967349 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.967310 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa194666-bb2c-4610-a051-7fd9caef77d3-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:23.968044 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:23.968016 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/154aa781-e85c-4aad-8fc1-e022e9704801-maas-api-service-ca" pod "154aa781-e85c-4aad-8fc1-e022e9704801" (UID: "154aa781-e85c-4aad-8fc1-e022e9704801"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:24.054959 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.054881 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154aa781-e85c-4aad-8fc1-e022e9704801-kube-api-access-q795d" pod "154aa781-e85c-4aad-8fc1-e022e9704801" (UID: "154aa781-e85c-4aad-8fc1-e022e9704801"). InnerVolumeSpecName "kube-api-access-q795d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:24.066894 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.066830 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa194666-bb2c-4610-a051-7fd9caef77d3-kube-api-access-g8pxg" pod "aa194666-bb2c-4610-a051-7fd9caef77d3" (UID: "aa194666-bb2c-4610-a051-7fd9caef77d3"). InnerVolumeSpecName "kube-api-access-g8pxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:24.071474 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.070878 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/154aa781-e85c-4aad-8fc1-e022e9704801-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:24.071474 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.070924 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q795d\" (UniqueName: \"kubernetes.io/projected/154aa781-e85c-4aad-8fc1-e022e9704801-kube-api-access-q795d\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:24.071474 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.070942 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8pxg\" (UniqueName: \"kubernetes.io/projected/aa194666-bb2c-4610-a051-7fd9caef77d3-kube-api-access-g8pxg\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:24.075527 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.075389 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx"] Nov 02 18:15:24.125298 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.125263 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r"] Nov 02 18:15:24.150201 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.150153 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7"] Nov 02 18:15:24.191551 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.191327 5601 patch_prober.go:29] interesting pod/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 02 18:15:24.191551 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.191383 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" podUID="84ed3fb1-d6b4-469c-b7a3-8af367d47208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 02 18:15:24.197956 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.197922 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg"] Nov 02 18:15:24.210106 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.210070 5601 generic.go:356] "Generic (PLEG): container finished" podID="3e296182-a14a-4ee7-9cd2-d014826b6cd9" containerID="c31127505e4f0a9d283607620299b193fe588ac8a0d0a76d24847cedb2edffaf" exitCode=60 Nov 02 18:15:24.210753 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.210212 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" event={"ID":"3e296182-a14a-4ee7-9cd2-d014826b6cd9","Type":"ContainerDied","Data":"c31127505e4f0a9d283607620299b193fe588ac8a0d0a76d24847cedb2edffaf"} Nov 02 18:15:24.210753 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.210270 5601 scope.go:122] "RemoveContainer" containerID="b167c10fdefbb5a742e1d4120dd9b5ce1ec74e490729512a4aead4327e4715be" Nov 02 18:15:24.271346 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.271303 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z" event={"ID":"e6aed869-903a-4e44-aaed-66d8940f4f90","Type":"ContainerDied","Data":"aa56341819cc5f9c6845f7b32fde80c6bc7cda1f69a6605c6c517f11f8c829a9"} Nov 02 18:15:24.271660 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.271638 5601 scope.go:122] "RemoveContainer" containerID="7d1baeda98d76b305bd8fa34066a748479866510f898237d065892512669db54" Nov 02 18:15:24.271993 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.271975 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z" Nov 02 18:15:24.428547 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.427752 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj"] Nov 02 18:15:24.474702 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.474596 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt" Nov 02 18:15:24.478434 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.478380 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt" event={"ID":"154aa781-e85c-4aad-8fc1-e022e9704801","Type":"ContainerDied","Data":"6266d96251ad1f9a773294f581a8eb5c38b75df4dc02e9c098232fee982d43be"} Nov 02 18:15:24.495844 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.495798 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" event={"ID":"bd2ceb92-645d-4255-89a8-c0211bc08025","Type":"ContainerStarted","Data":"90f6be9bf7e05c9e509beac89d07571e6737c749ac7f148c14c3480560c9b8cd"} Nov 02 18:15:24.506796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.506747 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s" event={"ID":"aa194666-bb2c-4610-a051-7fd9caef77d3","Type":"ContainerDied","Data":"71846bfab522d5f51d3a14b04387e76613f410230c6bed3b9d2ca576f99ad481"} Nov 02 18:15:24.507512 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.507244 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s" Nov 02 18:15:24.516778 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.516674 5601 scope.go:122] "RemoveContainer" containerID="1463ef2fe119059aba7e8b3bb7cb8078693bc36d8c69ac00255cc0a966fbba9d" Nov 02 18:15:24.531269 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.530834 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps"] Nov 02 18:15:24.581706 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.581424 5601 scope.go:122] "RemoveContainer" containerID="11209dcee18d14c142716ffbcb3f993d2c5634b68886103fd482a9995b34747d" Nov 02 18:15:24.581706 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.581503 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z"] Nov 02 18:15:24.588243 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.588217 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" Nov 02 18:15:24.600534 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.600500 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894055-5lb8z"] Nov 02 18:15:24.625507 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.625471 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt"] Nov 02 18:15:24.632679 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.632651 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894055-g5zgt"] Nov 02 18:15:24.657717 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.657672 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s"] Nov 02 18:15:24.660009 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.659607 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4e45226-6362-4b36-b890-11b6804c8d65-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e45226-6362-4b36-b890-11b6804c8d65-maas-api-service-ca\") pod \"c4e45226-6362-4b36-b890-11b6804c8d65\" (UID: \"c4e45226-6362-4b36-b890-11b6804c8d65\") " Nov 02 18:15:24.660668 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.660259 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e45226-6362-4b36-b890-11b6804c8d65-kube-api-access-p87rq\" (UniqueName: \"kubernetes.io/projected/c4e45226-6362-4b36-b890-11b6804c8d65-kube-api-access-p87rq\") pod \"c4e45226-6362-4b36-b890-11b6804c8d65\" (UID: \"c4e45226-6362-4b36-b890-11b6804c8d65\") " Nov 02 18:15:24.661159 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.661116 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e45226-6362-4b36-b890-11b6804c8d65-maas-api-service-ca" pod "c4e45226-6362-4b36-b890-11b6804c8d65" (UID: "c4e45226-6362-4b36-b890-11b6804c8d65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:24.662084 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.662001 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e45226-6362-4b36-b890-11b6804c8d65-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:24.666909 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.666842 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894055-t694s"] Nov 02 18:15:24.668891 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.668853 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e45226-6362-4b36-b890-11b6804c8d65-kube-api-access-p87rq" pod "c4e45226-6362-4b36-b890-11b6804c8d65" (UID: "c4e45226-6362-4b36-b890-11b6804c8d65"). InnerVolumeSpecName "kube-api-access-p87rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:24.715102 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.714752 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r" Nov 02 18:15:24.769861 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.769726 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b16fa692-4b98-42ad-b4db-b38d411c093a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16fa692-4b98-42ad-b4db-b38d411c093a-maas-api-service-ca\") pod \"b16fa692-4b98-42ad-b4db-b38d411c093a\" (UID: \"b16fa692-4b98-42ad-b4db-b38d411c093a\") " Nov 02 18:15:24.770039 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.769899 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b16fa692-4b98-42ad-b4db-b38d411c093a-kube-api-access-xn94r\" (UniqueName: \"kubernetes.io/projected/b16fa692-4b98-42ad-b4db-b38d411c093a-kube-api-access-xn94r\") pod \"b16fa692-4b98-42ad-b4db-b38d411c093a\" (UID: \"b16fa692-4b98-42ad-b4db-b38d411c093a\") " Nov 02 18:15:24.777362 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.770844 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p87rq\" (UniqueName: \"kubernetes.io/projected/c4e45226-6362-4b36-b890-11b6804c8d65-kube-api-access-p87rq\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:24.788383 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.787892 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b16fa692-4b98-42ad-b4db-b38d411c093a-maas-api-service-ca" pod "b16fa692-4b98-42ad-b4db-b38d411c093a" (UID: "b16fa692-4b98-42ad-b4db-b38d411c093a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:24.799543 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.799491 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16fa692-4b98-42ad-b4db-b38d411c093a-kube-api-access-xn94r" pod "b16fa692-4b98-42ad-b4db-b38d411c093a" (UID: "b16fa692-4b98-42ad-b4db-b38d411c093a"). InnerVolumeSpecName "kube-api-access-xn94r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:24.804722 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.804695 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg" Nov 02 18:15:24.808923 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.808896 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" Nov 02 18:15:24.871357 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.871324 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" Nov 02 18:15:24.871507 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.871432 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3611e836-f87d-448f-8f33-5f58724e18cf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611e836-f87d-448f-8f33-5f58724e18cf-maas-api-service-ca\") pod \"3611e836-f87d-448f-8f33-5f58724e18cf\" (UID: \"3611e836-f87d-448f-8f33-5f58724e18cf\") " Nov 02 18:15:24.871699 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.871674 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e296182-a14a-4ee7-9cd2-d014826b6cd9-kube-api-access-5b2mv\" (UniqueName: \"kubernetes.io/projected/3e296182-a14a-4ee7-9cd2-d014826b6cd9-kube-api-access-5b2mv\") pod \"3e296182-a14a-4ee7-9cd2-d014826b6cd9\" (UID: \"3e296182-a14a-4ee7-9cd2-d014826b6cd9\") " Nov 02 18:15:24.871835 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.871814 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e296182-a14a-4ee7-9cd2-d014826b6cd9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e296182-a14a-4ee7-9cd2-d014826b6cd9-maas-api-service-ca\") pod \"3e296182-a14a-4ee7-9cd2-d014826b6cd9\" (UID: \"3e296182-a14a-4ee7-9cd2-d014826b6cd9\") " Nov 02 18:15:24.871943 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.871924 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3611e836-f87d-448f-8f33-5f58724e18cf-kube-api-access-pz85d\" (UniqueName: \"kubernetes.io/projected/3611e836-f87d-448f-8f33-5f58724e18cf-kube-api-access-pz85d\") pod \"3611e836-f87d-448f-8f33-5f58724e18cf\" (UID: \"3611e836-f87d-448f-8f33-5f58724e18cf\") " Nov 02 18:15:24.872174 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.872147 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e296182-a14a-4ee7-9cd2-d014826b6cd9-maas-api-service-ca" pod "3e296182-a14a-4ee7-9cd2-d014826b6cd9" (UID: "3e296182-a14a-4ee7-9cd2-d014826b6cd9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:24.872293 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.871810 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3611e836-f87d-448f-8f33-5f58724e18cf-maas-api-service-ca" pod "3611e836-f87d-448f-8f33-5f58724e18cf" (UID: "3611e836-f87d-448f-8f33-5f58724e18cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:24.872764 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.872741 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e296182-a14a-4ee7-9cd2-d014826b6cd9-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:24.872764 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.872763 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn94r\" (UniqueName: \"kubernetes.io/projected/b16fa692-4b98-42ad-b4db-b38d411c093a-kube-api-access-xn94r\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:24.872922 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.872773 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611e836-f87d-448f-8f33-5f58724e18cf-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:24.872922 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.872782 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16fa692-4b98-42ad-b4db-b38d411c093a-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:24.875418 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.875393 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e296182-a14a-4ee7-9cd2-d014826b6cd9-kube-api-access-5b2mv" pod "3e296182-a14a-4ee7-9cd2-d014826b6cd9" (UID: "3e296182-a14a-4ee7-9cd2-d014826b6cd9"). InnerVolumeSpecName "kube-api-access-5b2mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:24.875601 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.875429 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" Nov 02 18:15:24.876524 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.876499 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3611e836-f87d-448f-8f33-5f58724e18cf-kube-api-access-pz85d" pod "3611e836-f87d-448f-8f33-5f58724e18cf" (UID: "3611e836-f87d-448f-8f33-5f58724e18cf"). InnerVolumeSpecName "kube-api-access-pz85d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:24.973325 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.973296 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/871bed77-cbe7-4001-a4f8-a6cedca05cef-kube-api-access-qvj9b\" (UniqueName: \"kubernetes.io/projected/871bed77-cbe7-4001-a4f8-a6cedca05cef-kube-api-access-qvj9b\") pod \"871bed77-cbe7-4001-a4f8-a6cedca05cef\" (UID: \"871bed77-cbe7-4001-a4f8-a6cedca05cef\") " Nov 02 18:15:24.973544 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.973525 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f91fe1-6e31-4c8c-a24e-2d473a179960-kube-api-access-mnkxw\" (UniqueName: \"kubernetes.io/projected/e2f91fe1-6e31-4c8c-a24e-2d473a179960-kube-api-access-mnkxw\") pod \"e2f91fe1-6e31-4c8c-a24e-2d473a179960\" (UID: \"e2f91fe1-6e31-4c8c-a24e-2d473a179960\") " Nov 02 18:15:24.973728 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.973705 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/871bed77-cbe7-4001-a4f8-a6cedca05cef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/871bed77-cbe7-4001-a4f8-a6cedca05cef-maas-api-service-ca\") pod \"871bed77-cbe7-4001-a4f8-a6cedca05cef\" (UID: \"871bed77-cbe7-4001-a4f8-a6cedca05cef\") " Nov 02 18:15:24.973818 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.973811 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2f91fe1-6e31-4c8c-a24e-2d473a179960-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f91fe1-6e31-4c8c-a24e-2d473a179960-maas-api-service-ca\") pod \"e2f91fe1-6e31-4c8c-a24e-2d473a179960\" (UID: \"e2f91fe1-6e31-4c8c-a24e-2d473a179960\") " Nov 02 18:15:24.974130 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.974101 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/871bed77-cbe7-4001-a4f8-a6cedca05cef-maas-api-service-ca" pod "871bed77-cbe7-4001-a4f8-a6cedca05cef" (UID: "871bed77-cbe7-4001-a4f8-a6cedca05cef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:24.974349 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.974326 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f91fe1-6e31-4c8c-a24e-2d473a179960-maas-api-service-ca" pod "e2f91fe1-6e31-4c8c-a24e-2d473a179960" (UID: "e2f91fe1-6e31-4c8c-a24e-2d473a179960"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:24.974446 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.974432 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz85d\" (UniqueName: \"kubernetes.io/projected/3611e836-f87d-448f-8f33-5f58724e18cf-kube-api-access-pz85d\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:24.974519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.974447 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/871bed77-cbe7-4001-a4f8-a6cedca05cef-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:24.974519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.974458 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f91fe1-6e31-4c8c-a24e-2d473a179960-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:24.974519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.974468 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b2mv\" (UniqueName: \"kubernetes.io/projected/3e296182-a14a-4ee7-9cd2-d014826b6cd9-kube-api-access-5b2mv\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:24.975655 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.975629 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871bed77-cbe7-4001-a4f8-a6cedca05cef-kube-api-access-qvj9b" pod "871bed77-cbe7-4001-a4f8-a6cedca05cef" (UID: "871bed77-cbe7-4001-a4f8-a6cedca05cef"). InnerVolumeSpecName "kube-api-access-qvj9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:24.975812 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:24.975793 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f91fe1-6e31-4c8c-a24e-2d473a179960-kube-api-access-mnkxw" pod "e2f91fe1-6e31-4c8c-a24e-2d473a179960" (UID: "e2f91fe1-6e31-4c8c-a24e-2d473a179960"). InnerVolumeSpecName "kube-api-access-mnkxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:25.076041 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.076002 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnkxw\" (UniqueName: \"kubernetes.io/projected/e2f91fe1-6e31-4c8c-a24e-2d473a179960-kube-api-access-mnkxw\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:25.076041 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.076038 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvj9b\" (UniqueName: \"kubernetes.io/projected/871bed77-cbe7-4001-a4f8-a6cedca05cef-kube-api-access-qvj9b\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:25.197678 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.197640 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:15:25.198193 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.198162 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:15:25.199055 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.199035 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:15:25.433641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.431788 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154aa781-e85c-4aad-8fc1-e022e9704801" path="/var/lib/kubelet/pods/154aa781-e85c-4aad-8fc1-e022e9704801/volumes" Nov 02 18:15:25.433641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.432866 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa194666-bb2c-4610-a051-7fd9caef77d3" path="/var/lib/kubelet/pods/aa194666-bb2c-4610-a051-7fd9caef77d3/volumes" Nov 02 18:15:25.433641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.433438 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6aed869-903a-4e44-aaed-66d8940f4f90" path="/var/lib/kubelet/pods/e6aed869-903a-4e44-aaed-66d8940f4f90/volumes" Nov 02 18:15:25.640468 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.632182 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj"] Nov 02 18:15:25.688719 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.675859 5601 generic.go:356] "Generic (PLEG): container finished" podID="bd2ceb92-645d-4255-89a8-c0211bc08025" containerID="90f6be9bf7e05c9e509beac89d07571e6737c749ac7f148c14c3480560c9b8cd" exitCode=60 Nov 02 18:15:25.688719 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.675963 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" event={"ID":"bd2ceb92-645d-4255-89a8-c0211bc08025","Type":"ContainerDied","Data":"90f6be9bf7e05c9e509beac89d07571e6737c749ac7f148c14c3480560c9b8cd"} Nov 02 18:15:25.688719 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.676023 5601 scope.go:122] "RemoveContainer" containerID="a70ab00820a19e33994b167d3b6ed533c22b73a5d75eb371e728670d1df11898" Nov 02 18:15:25.890650 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.878531 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" event={"ID":"c4e45226-6362-4b36-b890-11b6804c8d65","Type":"ContainerDied","Data":"638b66edb805930e04d92bd2929abea330ff9d976a6e9e92c6ae49aeb8a155b1"} Nov 02 18:15:25.906739 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.891703 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx" Nov 02 18:15:25.924816 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.901902 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r" event={"ID":"b16fa692-4b98-42ad-b4db-b38d411c093a","Type":"ContainerDied","Data":"fa26bd902d87483c89b45e9b7c0092b0831f393c2bea0a757301838dc0bc895a"} Nov 02 18:15:25.924816 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.902022 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r" Nov 02 18:15:25.925033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.924891 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" event={"ID":"3e296182-a14a-4ee7-9cd2-d014826b6cd9","Type":"ContainerDied","Data":"0e85d0d33ac5be12bde3360d8fec44c4cac6ac6a64f88c765d5d44a19e4af4c9"} Nov 02 18:15:25.925087 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.925035 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7" Nov 02 18:15:25.938602 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.937291 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg" event={"ID":"3611e836-f87d-448f-8f33-5f58724e18cf","Type":"ContainerDied","Data":"7fc6ae65a3d51b510a50eec6d4eaa3f30e9bce1352afb23233d421a5512d3377"} Nov 02 18:15:25.940964 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.938805 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg" Nov 02 18:15:25.946104 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.946071 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" event={"ID":"e2f91fe1-6e31-4c8c-a24e-2d473a179960","Type":"ContainerDied","Data":"6f627a35a5e5cad3610b7e90278495c59c79e6a278090c690e07f361a44a36ca"} Nov 02 18:15:25.951682 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.949321 5601 scope.go:122] "RemoveContainer" containerID="6ac2da68f625a06efc6c151d7e2add962d3378328248a9b1ab34565dcf1c2114" Nov 02 18:15:25.957514 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.957475 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj" Nov 02 18:15:25.966119 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.966094 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" Nov 02 18:15:25.966512 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.966147 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps" event={"ID":"871bed77-cbe7-4001-a4f8-a6cedca05cef","Type":"ContainerDied","Data":"31935b9fc170ab9708e683a341de5607561919f68ff92e8128c24b0c41eea579"} Nov 02 18:15:25.979180 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.978185 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r"] Nov 02 18:15:25.985697 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.985672 5601 scope.go:122] "RemoveContainer" containerID="7e82c16a766f2d4cbc9c540655976f65523cfe703a2cc13784c9a1ea74bd5a26" Nov 02 18:15:25.992940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.992793 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894055-cq62r"] Nov 02 18:15:25.999752 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:25.999696 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7"] Nov 02 18:15:26.001802 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.000610 5601 scope.go:122] "RemoveContainer" containerID="c31127505e4f0a9d283607620299b193fe588ac8a0d0a76d24847cedb2edffaf" Nov 02 18:15:26.008694 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.008556 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-658b8c-29894055-zbtw7"] Nov 02 18:15:26.016845 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.016824 5601 scope.go:122] "RemoveContainer" containerID="8184486db8cb724f33026c7336b196be283fb4a3f200585a9465e9a00fff7a2d" Nov 02 18:15:26.022441 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.022420 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx"] Nov 02 18:15:26.029997 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.027818 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b6b6f4-29894055-lhfkx"] Nov 02 18:15:26.033238 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.033177 5601 scope.go:122] "RemoveContainer" containerID="cc0166be9a68eb70d8468a0d9a04086bc0f0ef3434b946ec20da7f1beb065499" Nov 02 18:15:26.047419 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.047389 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps"] Nov 02 18:15:26.053363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.053313 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0f0baf-29894055-m8sps"] Nov 02 18:15:26.064357 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.064332 5601 scope.go:122] "RemoveContainer" containerID="b98049a97fc1752ec230ebe0fee5b4536cfc6a8a991d3713886b70cc4d7562e5" Nov 02 18:15:26.075568 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.075540 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg"] Nov 02 18:15:26.077838 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.077813 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894055-b7xpg"] Nov 02 18:15:26.088773 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.088705 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj"] Nov 02 18:15:26.104482 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.104433 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894055-c8mxj"] Nov 02 18:15:26.191822 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.191734 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" Nov 02 18:15:26.364303 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.364246 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd2ceb92-645d-4255-89a8-c0211bc08025-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2ceb92-645d-4255-89a8-c0211bc08025-maas-api-service-ca\") pod \"bd2ceb92-645d-4255-89a8-c0211bc08025\" (UID: \"bd2ceb92-645d-4255-89a8-c0211bc08025\") " Nov 02 18:15:26.364519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.364498 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2ceb92-645d-4255-89a8-c0211bc08025-kube-api-access-rrqjq\" (UniqueName: \"kubernetes.io/projected/bd2ceb92-645d-4255-89a8-c0211bc08025-kube-api-access-rrqjq\") pod \"bd2ceb92-645d-4255-89a8-c0211bc08025\" (UID: \"bd2ceb92-645d-4255-89a8-c0211bc08025\") " Nov 02 18:15:26.364786 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.364755 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2ceb92-645d-4255-89a8-c0211bc08025-maas-api-service-ca" pod "bd2ceb92-645d-4255-89a8-c0211bc08025" (UID: "bd2ceb92-645d-4255-89a8-c0211bc08025"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:26.365293 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.365257 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2ceb92-645d-4255-89a8-c0211bc08025-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:26.367366 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.367336 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2ceb92-645d-4255-89a8-c0211bc08025-kube-api-access-rrqjq" pod "bd2ceb92-645d-4255-89a8-c0211bc08025" (UID: "bd2ceb92-645d-4255-89a8-c0211bc08025"). InnerVolumeSpecName "kube-api-access-rrqjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:26.466859 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.466770 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrqjq\" (UniqueName: \"kubernetes.io/projected/bd2ceb92-645d-4255-89a8-c0211bc08025-kube-api-access-rrqjq\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:26.978626 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.978582 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" Nov 02 18:15:26.978806 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.978611 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj" event={"ID":"bd2ceb92-645d-4255-89a8-c0211bc08025","Type":"ContainerDied","Data":"131a871760b35b17f7692044161936ce75f7ee543039c887f0ad7b7cfb006e24"} Nov 02 18:15:26.978806 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:26.978708 5601 scope.go:122] "RemoveContainer" containerID="90f6be9bf7e05c9e509beac89d07571e6737c749ac7f148c14c3480560c9b8cd" Nov 02 18:15:27.007004 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.006972 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj"] Nov 02 18:15:27.012578 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.012552 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-78383f-29894055-rgdzj"] Nov 02 18:15:27.433175 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.433141 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3611e836-f87d-448f-8f33-5f58724e18cf" path="/var/lib/kubelet/pods/3611e836-f87d-448f-8f33-5f58724e18cf/volumes" Nov 02 18:15:27.433706 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.433689 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e296182-a14a-4ee7-9cd2-d014826b6cd9" path="/var/lib/kubelet/pods/3e296182-a14a-4ee7-9cd2-d014826b6cd9/volumes" Nov 02 18:15:27.434121 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.434106 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="871bed77-cbe7-4001-a4f8-a6cedca05cef" path="/var/lib/kubelet/pods/871bed77-cbe7-4001-a4f8-a6cedca05cef/volumes" Nov 02 18:15:27.435060 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.435036 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b16fa692-4b98-42ad-b4db-b38d411c093a" path="/var/lib/kubelet/pods/b16fa692-4b98-42ad-b4db-b38d411c093a/volumes" Nov 02 18:15:27.435722 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.435703 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd2ceb92-645d-4255-89a8-c0211bc08025" path="/var/lib/kubelet/pods/bd2ceb92-645d-4255-89a8-c0211bc08025/volumes" Nov 02 18:15:27.436334 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.436313 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4e45226-6362-4b36-b890-11b6804c8d65" path="/var/lib/kubelet/pods/c4e45226-6362-4b36-b890-11b6804c8d65/volumes" Nov 02 18:15:27.437211 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.437190 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f91fe1-6e31-4c8c-a24e-2d473a179960" path="/var/lib/kubelet/pods/e2f91fe1-6e31-4c8c-a24e-2d473a179960/volumes" Nov 02 18:15:27.528031 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.527981 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b"] Nov 02 18:15:27.529122 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529093 5601 cpu_manager.go:414] "containerMap: removing container" podUID="b16fa692-4b98-42ad-b4db-b38d411c093a" containerName="cleanup" Nov 02 18:15:27.529122 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529121 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16fa692-4b98-42ad-b4db-b38d411c093a" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529135 5601 cpu_manager.go:414] "containerMap: removing container" podUID="62588dd0-b7ee-440e-ab7e-419d7429b2d9" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529144 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62588dd0-b7ee-440e-ab7e-419d7429b2d9" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529166 5601 cpu_manager.go:414] "containerMap: removing container" podUID="e2f91fe1-6e31-4c8c-a24e-2d473a179960" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529176 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f91fe1-6e31-4c8c-a24e-2d473a179960" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529188 5601 cpu_manager.go:414] "containerMap: removing container" podUID="3e296182-a14a-4ee7-9cd2-d014826b6cd9" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529195 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e296182-a14a-4ee7-9cd2-d014826b6cd9" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529206 5601 cpu_manager.go:414] "containerMap: removing container" podUID="bd2ceb92-645d-4255-89a8-c0211bc08025" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529213 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2ceb92-645d-4255-89a8-c0211bc08025" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529224 5601 cpu_manager.go:414] "containerMap: removing container" podUID="871bed77-cbe7-4001-a4f8-a6cedca05cef" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529232 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="871bed77-cbe7-4001-a4f8-a6cedca05cef" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529245 5601 cpu_manager.go:414] "containerMap: removing container" podUID="18f2b645-9cdf-4b3e-8b54-8e31706d33a3" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529252 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18f2b645-9cdf-4b3e-8b54-8e31706d33a3" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529261 5601 cpu_manager.go:414] "containerMap: removing container" podUID="62588dd0-b7ee-440e-ab7e-419d7429b2d9" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529268 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62588dd0-b7ee-440e-ab7e-419d7429b2d9" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529278 5601 cpu_manager.go:414] "containerMap: removing container" podUID="3e296182-a14a-4ee7-9cd2-d014826b6cd9" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529285 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e296182-a14a-4ee7-9cd2-d014826b6cd9" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529301 5601 cpu_manager.go:414] "containerMap: removing container" podUID="154aa781-e85c-4aad-8fc1-e022e9704801" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529308 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154aa781-e85c-4aad-8fc1-e022e9704801" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529321 5601 cpu_manager.go:414] "containerMap: removing container" podUID="23af2a44-3074-4dce-a5c6-9c18b4c87d35" containerName="cleanup" Nov 02 18:15:27.529318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529328 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23af2a44-3074-4dce-a5c6-9c18b4c87d35" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529336 5601 cpu_manager.go:414] "containerMap: removing container" podUID="3611e836-f87d-448f-8f33-5f58724e18cf" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529344 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3611e836-f87d-448f-8f33-5f58724e18cf" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529353 5601 cpu_manager.go:414] "containerMap: removing container" podUID="6612ae11-596f-4056-b6bc-42ed18333072" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529360 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6612ae11-596f-4056-b6bc-42ed18333072" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529377 5601 cpu_manager.go:414] "containerMap: removing container" podUID="c4e45226-6362-4b36-b890-11b6804c8d65" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529384 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e45226-6362-4b36-b890-11b6804c8d65" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529404 5601 cpu_manager.go:414] "containerMap: removing container" podUID="bd2ceb92-645d-4255-89a8-c0211bc08025" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529411 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2ceb92-645d-4255-89a8-c0211bc08025" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529425 5601 cpu_manager.go:414] "containerMap: removing container" podUID="3e296182-a14a-4ee7-9cd2-d014826b6cd9" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529433 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e296182-a14a-4ee7-9cd2-d014826b6cd9" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529446 5601 cpu_manager.go:414] "containerMap: removing container" podUID="154aa781-e85c-4aad-8fc1-e022e9704801" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529453 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154aa781-e85c-4aad-8fc1-e022e9704801" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529467 5601 cpu_manager.go:414] "containerMap: removing container" podUID="3611e836-f87d-448f-8f33-5f58724e18cf" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529474 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3611e836-f87d-448f-8f33-5f58724e18cf" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529487 5601 cpu_manager.go:414] "containerMap: removing container" podUID="23af2a44-3074-4dce-a5c6-9c18b4c87d35" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529494 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23af2a44-3074-4dce-a5c6-9c18b4c87d35" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529509 5601 cpu_manager.go:414] "containerMap: removing container" podUID="18f2b645-9cdf-4b3e-8b54-8e31706d33a3" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529516 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18f2b645-9cdf-4b3e-8b54-8e31706d33a3" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529524 5601 cpu_manager.go:414] "containerMap: removing container" podUID="c4e45226-6362-4b36-b890-11b6804c8d65" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529533 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e45226-6362-4b36-b890-11b6804c8d65" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529545 5601 cpu_manager.go:414] "containerMap: removing container" podUID="bd2ceb92-645d-4255-89a8-c0211bc08025" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529552 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2ceb92-645d-4255-89a8-c0211bc08025" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529569 5601 cpu_manager.go:414] "containerMap: removing container" podUID="b16fa692-4b98-42ad-b4db-b38d411c093a" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529576 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16fa692-4b98-42ad-b4db-b38d411c093a" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529591 5601 cpu_manager.go:414] "containerMap: removing container" podUID="aa194666-bb2c-4610-a051-7fd9caef77d3" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529598 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa194666-bb2c-4610-a051-7fd9caef77d3" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529607 5601 cpu_manager.go:414] "containerMap: removing container" podUID="6612ae11-596f-4056-b6bc-42ed18333072" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529630 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6612ae11-596f-4056-b6bc-42ed18333072" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529640 5601 cpu_manager.go:414] "containerMap: removing container" podUID="e6aed869-903a-4e44-aaed-66d8940f4f90" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529647 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6aed869-903a-4e44-aaed-66d8940f4f90" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529658 5601 cpu_manager.go:414] "containerMap: removing container" podUID="c4e45226-6362-4b36-b890-11b6804c8d65" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529664 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e45226-6362-4b36-b890-11b6804c8d65" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529681 5601 cpu_manager.go:414] "containerMap: removing container" podUID="aa194666-bb2c-4610-a051-7fd9caef77d3" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529689 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa194666-bb2c-4610-a051-7fd9caef77d3" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529706 5601 cpu_manager.go:414] "containerMap: removing container" podUID="e2f91fe1-6e31-4c8c-a24e-2d473a179960" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529713 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f91fe1-6e31-4c8c-a24e-2d473a179960" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529725 5601 cpu_manager.go:414] "containerMap: removing container" podUID="871bed77-cbe7-4001-a4f8-a6cedca05cef" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529731 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="871bed77-cbe7-4001-a4f8-a6cedca05cef" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529747 5601 cpu_manager.go:414] "containerMap: removing container" podUID="e6aed869-903a-4e44-aaed-66d8940f4f90" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529754 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6aed869-903a-4e44-aaed-66d8940f4f90" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529769 5601 cpu_manager.go:414] "containerMap: removing container" podUID="e6aed869-903a-4e44-aaed-66d8940f4f90" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529776 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6aed869-903a-4e44-aaed-66d8940f4f90" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529938 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="23af2a44-3074-4dce-a5c6-9c18b4c87d35" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529950 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="62588dd0-b7ee-440e-ab7e-419d7429b2d9" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529962 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="18f2b645-9cdf-4b3e-8b54-8e31706d33a3" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529972 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16fa692-4b98-42ad-b4db-b38d411c093a" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529980 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="6612ae11-596f-4056-b6bc-42ed18333072" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529990 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e45226-6362-4b36-b890-11b6804c8d65" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.529999 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2ceb92-645d-4255-89a8-c0211bc08025" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530009 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16fa692-4b98-42ad-b4db-b38d411c093a" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530020 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa194666-bb2c-4610-a051-7fd9caef77d3" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530028 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="6612ae11-596f-4056-b6bc-42ed18333072" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530036 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="154aa781-e85c-4aad-8fc1-e022e9704801" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530045 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6aed869-903a-4e44-aaed-66d8940f4f90" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530055 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e45226-6362-4b36-b890-11b6804c8d65" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530062 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f91fe1-6e31-4c8c-a24e-2d473a179960" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530074 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f91fe1-6e31-4c8c-a24e-2d473a179960" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530083 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="871bed77-cbe7-4001-a4f8-a6cedca05cef" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530091 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="871bed77-cbe7-4001-a4f8-a6cedca05cef" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530101 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6aed869-903a-4e44-aaed-66d8940f4f90" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530111 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa194666-bb2c-4610-a051-7fd9caef77d3" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530119 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6aed869-903a-4e44-aaed-66d8940f4f90" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530130 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="62588dd0-b7ee-440e-ab7e-419d7429b2d9" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530140 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e296182-a14a-4ee7-9cd2-d014826b6cd9" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530150 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="62588dd0-b7ee-440e-ab7e-419d7429b2d9" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530158 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e296182-a14a-4ee7-9cd2-d014826b6cd9" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530169 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="154aa781-e85c-4aad-8fc1-e022e9704801" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530177 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="23af2a44-3074-4dce-a5c6-9c18b4c87d35" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530186 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="3611e836-f87d-448f-8f33-5f58724e18cf" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530197 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2ceb92-645d-4255-89a8-c0211bc08025" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530206 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e296182-a14a-4ee7-9cd2-d014826b6cd9" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530215 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="18f2b645-9cdf-4b3e-8b54-8e31706d33a3" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530224 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="3611e836-f87d-448f-8f33-5f58724e18cf" containerName="cleanup" Nov 02 18:15:27.530363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.530233 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="3611e836-f87d-448f-8f33-5f58724e18cf" containerName="cleanup" Nov 02 18:15:27.535713 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.535686 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.547497 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.547467 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-22dfc1-openshift-default-dockercfg-c7p8d\"" Nov 02 18:15:27.557421 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.557380 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b"] Nov 02 18:15:27.585949 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.585903 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-podinfo\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.586243 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.586050 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-envoy\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.586243 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.586103 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnkrh\" (UniqueName: \"kubernetes.io/projected/208dcda2-8e4e-4f47-af69-942e7412eda9-kube-api-access-gnkrh\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.586243 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.586167 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-credential-socket\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.586243 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.586195 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-workload-certs\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.586608 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.586244 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-workload-socket\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.586608 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.586278 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-token\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.586608 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.586412 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208dcda2-8e4e-4f47-af69-942e7412eda9-istiod-ca-cert\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.586608 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.586517 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-data\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.736691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.699001 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208dcda2-8e4e-4f47-af69-942e7412eda9-istiod-ca-cert\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.736691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.706753 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-data\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.736691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.706828 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-podinfo\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.736691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.706969 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-envoy\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.736691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.707027 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnkrh\" (UniqueName: \"kubernetes.io/projected/208dcda2-8e4e-4f47-af69-942e7412eda9-kube-api-access-gnkrh\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.736691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.707117 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-credential-socket\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.736691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.707150 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-workload-certs\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.736691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.707211 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-workload-socket\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.736691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.707247 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-token\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.736691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.708422 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208dcda2-8e4e-4f47-af69-942e7412eda9-istiod-ca-cert\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.736691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.708723 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-data\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.736691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.708963 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-credential-socket\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.736691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.709192 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-workload-certs\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.736691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.709432 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-workload-socket\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.736691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.720924 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-envoy\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.780977 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.780907 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnkrh\" (UniqueName: \"kubernetes.io/projected/208dcda2-8e4e-4f47-af69-942e7412eda9-kube-api-access-gnkrh\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.782820 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.782745 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-podinfo\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.825714 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.824896 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-token\") pod \"e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.874327 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.873942 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:27.923254 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:27.921851 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" Nov 02 18:15:28.371146 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:28.371110 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b"] Nov 02 18:15:28.373659 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:15:28.373585 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod208dcda2_8e4e_4f47_af69_942e7412eda9.slice/crio-25d2ca49e62571a2c1c3a622cb9bf48edfd2f16474b7d374ae745d71f176408d WatchSource:0}: Error finding container 25d2ca49e62571a2c1c3a622cb9bf48edfd2f16474b7d374ae745d71f176408d: Status 404 returned error can't find the container with id 25d2ca49e62571a2c1c3a622cb9bf48edfd2f16474b7d374ae745d71f176408d Nov 02 18:15:28.377832 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:28.377788 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:15:28.377953 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:28.377873 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:15:28.377953 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:28.377934 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:15:29.052908 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:29.052862 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" event={"ID":"208dcda2-8e4e-4f47-af69-942e7412eda9","Type":"ContainerStarted","Data":"b946e58b3b2011581fc25302f71f1f739853e738a3c262b9eb7833e3307b6d19"} Nov 02 18:15:29.053404 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:29.052944 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" event={"ID":"208dcda2-8e4e-4f47-af69-942e7412eda9","Type":"ContainerStarted","Data":"25d2ca49e62571a2c1c3a622cb9bf48edfd2f16474b7d374ae745d71f176408d"} Nov 02 18:15:29.079226 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:29.079166 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" podStartSLOduration=2.079152974 podStartE2EDuration="2.079152974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:29.073065658 +0000 UTC m=+2493.764119695" watchObservedRunningTime="2026-11-02 18:15:29.079152974 +0000 UTC m=+2493.770206985" Nov 02 18:15:29.876517 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:29.876474 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:29.883943 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:29.883912 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:30.062110 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:30.062063 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:30.063247 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:30.063227 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:15:32.647986 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.647949 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5"] Nov 02 18:15:32.648750 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648731 5601 cpu_manager.go:414] "containerMap: removing container" podUID="e2f91fe1-6e31-4c8c-a24e-2d473a179960" containerName="cleanup" Nov 02 18:15:32.648750 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648748 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f91fe1-6e31-4c8c-a24e-2d473a179960" containerName="cleanup" Nov 02 18:15:32.648940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648762 5601 cpu_manager.go:414] "containerMap: removing container" podUID="871bed77-cbe7-4001-a4f8-a6cedca05cef" containerName="cleanup" Nov 02 18:15:32.648940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648767 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="871bed77-cbe7-4001-a4f8-a6cedca05cef" containerName="cleanup" Nov 02 18:15:32.648940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648776 5601 cpu_manager.go:414] "containerMap: removing container" podUID="18f2b645-9cdf-4b3e-8b54-8e31706d33a3" containerName="cleanup" Nov 02 18:15:32.648940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648781 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18f2b645-9cdf-4b3e-8b54-8e31706d33a3" containerName="cleanup" Nov 02 18:15:32.648940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648787 5601 cpu_manager.go:414] "containerMap: removing container" podUID="62588dd0-b7ee-440e-ab7e-419d7429b2d9" containerName="cleanup" Nov 02 18:15:32.648940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648791 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62588dd0-b7ee-440e-ab7e-419d7429b2d9" containerName="cleanup" Nov 02 18:15:32.648940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648801 5601 cpu_manager.go:414] "containerMap: removing container" podUID="154aa781-e85c-4aad-8fc1-e022e9704801" containerName="cleanup" Nov 02 18:15:32.648940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648805 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154aa781-e85c-4aad-8fc1-e022e9704801" containerName="cleanup" Nov 02 18:15:32.648940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648815 5601 cpu_manager.go:414] "containerMap: removing container" podUID="6612ae11-596f-4056-b6bc-42ed18333072" containerName="cleanup" Nov 02 18:15:32.648940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648821 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6612ae11-596f-4056-b6bc-42ed18333072" containerName="cleanup" Nov 02 18:15:32.648940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648850 5601 cpu_manager.go:414] "containerMap: removing container" podUID="3611e836-f87d-448f-8f33-5f58724e18cf" containerName="cleanup" Nov 02 18:15:32.648940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648854 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3611e836-f87d-448f-8f33-5f58724e18cf" containerName="cleanup" Nov 02 18:15:32.648940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648861 5601 cpu_manager.go:414] "containerMap: removing container" podUID="23af2a44-3074-4dce-a5c6-9c18b4c87d35" containerName="cleanup" Nov 02 18:15:32.648940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648866 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23af2a44-3074-4dce-a5c6-9c18b4c87d35" containerName="cleanup" Nov 02 18:15:32.648940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648881 5601 cpu_manager.go:414] "containerMap: removing container" podUID="b16fa692-4b98-42ad-b4db-b38d411c093a" containerName="cleanup" Nov 02 18:15:32.648940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648887 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16fa692-4b98-42ad-b4db-b38d411c093a" containerName="cleanup" Nov 02 18:15:32.648940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648899 5601 cpu_manager.go:414] "containerMap: removing container" podUID="aa194666-bb2c-4610-a051-7fd9caef77d3" containerName="cleanup" Nov 02 18:15:32.648940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.648904 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa194666-bb2c-4610-a051-7fd9caef77d3" containerName="cleanup" Nov 02 18:15:32.649912 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.649011 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2ceb92-645d-4255-89a8-c0211bc08025" containerName="cleanup" Nov 02 18:15:32.649912 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.649021 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="18f2b645-9cdf-4b3e-8b54-8e31706d33a3" containerName="cleanup" Nov 02 18:15:32.649912 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.649030 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="23af2a44-3074-4dce-a5c6-9c18b4c87d35" containerName="cleanup" Nov 02 18:15:32.649912 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.649037 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e45226-6362-4b36-b890-11b6804c8d65" containerName="cleanup" Nov 02 18:15:32.649912 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.649044 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16fa692-4b98-42ad-b4db-b38d411c093a" containerName="cleanup" Nov 02 18:15:32.649912 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.649050 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa194666-bb2c-4610-a051-7fd9caef77d3" containerName="cleanup" Nov 02 18:15:32.649912 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.649057 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="6612ae11-596f-4056-b6bc-42ed18333072" containerName="cleanup" Nov 02 18:15:32.649912 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.649063 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="154aa781-e85c-4aad-8fc1-e022e9704801" containerName="cleanup" Nov 02 18:15:32.649912 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.649069 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f91fe1-6e31-4c8c-a24e-2d473a179960" containerName="cleanup" Nov 02 18:15:32.649912 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.649074 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="871bed77-cbe7-4001-a4f8-a6cedca05cef" containerName="cleanup" Nov 02 18:15:32.654269 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.654249 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" Nov 02 18:15:32.661050 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.661026 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5"] Nov 02 18:15:32.771648 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.749125 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-22dfc1-dockercfg-rlrqw\"" Nov 02 18:15:32.830642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.771766 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-22dfc1\"" Nov 02 18:15:32.830642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.781206 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsvzt\" (UniqueName: \"kubernetes.io/projected/64765539-809c-42a0-af1c-7d1e6b506b18-kube-api-access-gsvzt\") pod \"payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5\" (UID: \"64765539-809c-42a0-af1c-7d1e6b506b18\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" Nov 02 18:15:32.830642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.828047 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64765539-809c-42a0-af1c-7d1e6b506b18-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5\" (UID: \"64765539-809c-42a0-af1c-7d1e6b506b18\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" Nov 02 18:15:32.977638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.960242 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsvzt\" (UniqueName: \"kubernetes.io/projected/64765539-809c-42a0-af1c-7d1e6b506b18-kube-api-access-gsvzt\") pod \"payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5\" (UID: \"64765539-809c-42a0-af1c-7d1e6b506b18\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" Nov 02 18:15:32.977638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.960487 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64765539-809c-42a0-af1c-7d1e6b506b18-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5\" (UID: \"64765539-809c-42a0-af1c-7d1e6b506b18\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" Nov 02 18:15:32.977638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:32.961573 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64765539-809c-42a0-af1c-7d1e6b506b18-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5\" (UID: \"64765539-809c-42a0-af1c-7d1e6b506b18\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" Nov 02 18:15:33.042438 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:33.034950 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsvzt\" (UniqueName: \"kubernetes.io/projected/64765539-809c-42a0-af1c-7d1e6b506b18-kube-api-access-gsvzt\") pod \"payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5\" (UID: \"64765539-809c-42a0-af1c-7d1e6b506b18\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" Nov 02 18:15:33.274687 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:33.274584 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" Nov 02 18:15:33.566151 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:33.566124 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5"] Nov 02 18:15:33.568753 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:15:33.568720 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64765539_809c_42a0_af1c_7d1e6b506b18.slice/crio-efcc3a15c9244c4cdb23c87363740a3b328cabf56386e6ba1f59a2b460857cd1 WatchSource:0}: Error finding container efcc3a15c9244c4cdb23c87363740a3b328cabf56386e6ba1f59a2b460857cd1: Status 404 returned error can't find the container with id efcc3a15c9244c4cdb23c87363740a3b328cabf56386e6ba1f59a2b460857cd1 Nov 02 18:15:34.115796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:34.115751 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" event={"ID":"64765539-809c-42a0-af1c-7d1e6b506b18","Type":"ContainerStarted","Data":"1090fa61dc0cbe07ede96e593d5cdeb79c8b0dbc9b256f55a24fbf0840657ef5"} Nov 02 18:15:34.116336 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:34.115820 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" event={"ID":"64765539-809c-42a0-af1c-7d1e6b506b18","Type":"ContainerStarted","Data":"efcc3a15c9244c4cdb23c87363740a3b328cabf56386e6ba1f59a2b460857cd1"} Nov 02 18:15:34.116336 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:34.115983 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" Nov 02 18:15:34.166707 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:34.164247 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" podStartSLOduration=2.164223935 podStartE2EDuration="2.164223935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:34.144334821 +0000 UTC m=+2498.835388834" watchObservedRunningTime="2026-11-02 18:15:34.164223935 +0000 UTC m=+2498.855277949" Nov 02 18:15:34.485833 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:34.485793 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:15:45.140927 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:45.140889 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" Nov 02 18:15:52.013080 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:52.013042 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch"] Nov 02 18:15:52.019919 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:52.019893 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" Nov 02 18:15:52.030775 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:52.028255 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch"] Nov 02 18:15:52.030775 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:52.028488 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-22dfc1-serving-cert\"" Nov 02 18:15:52.175381 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:52.175293 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/851b2b14-e25a-47b9-8eab-0e68096e6268-maas-api-tls\") pod \"maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch\" (UID: \"851b2b14-e25a-47b9-8eab-0e68096e6268\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" Nov 02 18:15:52.175935 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:52.175905 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh7jv\" (UniqueName: \"kubernetes.io/projected/851b2b14-e25a-47b9-8eab-0e68096e6268-kube-api-access-nh7jv\") pod \"maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch\" (UID: \"851b2b14-e25a-47b9-8eab-0e68096e6268\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" Nov 02 18:15:52.176724 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:52.176700 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/851b2b14-e25a-47b9-8eab-0e68096e6268-openshift-service-ca\") pod \"maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch\" (UID: \"851b2b14-e25a-47b9-8eab-0e68096e6268\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" Nov 02 18:15:52.177060 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:52.177009 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/851b2b14-e25a-47b9-8eab-0e68096e6268-metrics-certs\") pod \"maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch\" (UID: \"851b2b14-e25a-47b9-8eab-0e68096e6268\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" Nov 02 18:15:52.279250 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:52.279155 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/851b2b14-e25a-47b9-8eab-0e68096e6268-openshift-service-ca\") pod \"maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch\" (UID: \"851b2b14-e25a-47b9-8eab-0e68096e6268\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" Nov 02 18:15:52.279250 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:52.279215 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/851b2b14-e25a-47b9-8eab-0e68096e6268-metrics-certs\") pod \"maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch\" (UID: \"851b2b14-e25a-47b9-8eab-0e68096e6268\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" Nov 02 18:15:52.279464 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:52.279420 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/851b2b14-e25a-47b9-8eab-0e68096e6268-maas-api-tls\") pod \"maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch\" (UID: \"851b2b14-e25a-47b9-8eab-0e68096e6268\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" Nov 02 18:15:52.279512 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:52.279482 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh7jv\" (UniqueName: \"kubernetes.io/projected/851b2b14-e25a-47b9-8eab-0e68096e6268-kube-api-access-nh7jv\") pod \"maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch\" (UID: \"851b2b14-e25a-47b9-8eab-0e68096e6268\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" Nov 02 18:15:52.279579 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:52.279554 5601 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-serving-cert: secret "maas-api-e2e-inf-a-22dfc1-serving-cert" not found Nov 02 18:15:52.279672 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:15:52.279659 5601 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/851b2b14-e25a-47b9-8eab-0e68096e6268-maas-api-tls podName:851b2b14-e25a-47b9-8eab-0e68096e6268 nodeName:}" failed. No retries permitted until 2026-11-02 18:15:52.779638986 +0000 UTC m=+2517.470693018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/851b2b14-e25a-47b9-8eab-0e68096e6268-maas-api-tls") pod "maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" (UID: "851b2b14-e25a-47b9-8eab-0e68096e6268") : secret "maas-api-e2e-inf-a-22dfc1-serving-cert" not found Nov 02 18:15:52.279983 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:52.279965 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/851b2b14-e25a-47b9-8eab-0e68096e6268-openshift-service-ca\") pod \"maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch\" (UID: \"851b2b14-e25a-47b9-8eab-0e68096e6268\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" Nov 02 18:15:52.282056 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:52.282026 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/851b2b14-e25a-47b9-8eab-0e68096e6268-metrics-certs\") pod \"maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch\" (UID: \"851b2b14-e25a-47b9-8eab-0e68096e6268\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" Nov 02 18:15:52.288116 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:52.288091 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh7jv\" (UniqueName: \"kubernetes.io/projected/851b2b14-e25a-47b9-8eab-0e68096e6268-kube-api-access-nh7jv\") pod \"maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch\" (UID: \"851b2b14-e25a-47b9-8eab-0e68096e6268\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" Nov 02 18:15:52.808264 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:52.808228 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/851b2b14-e25a-47b9-8eab-0e68096e6268-maas-api-tls\") pod \"maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch\" (UID: \"851b2b14-e25a-47b9-8eab-0e68096e6268\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" Nov 02 18:15:52.810775 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:52.810749 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/851b2b14-e25a-47b9-8eab-0e68096e6268-maas-api-tls\") pod \"maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch\" (UID: \"851b2b14-e25a-47b9-8eab-0e68096e6268\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" Nov 02 18:15:52.954882 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:52.954851 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" Nov 02 18:15:53.301870 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:53.301826 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch"] Nov 02 18:15:53.302303 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:15:53.301981 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851b2b14_e25a_47b9_8eab_0e68096e6268.slice/crio-0d176df2dd262c6d63ad22962fe79cc714b0aac81a676d52fd394f8e8c6a8612 WatchSource:0}: Error finding container 0d176df2dd262c6d63ad22962fe79cc714b0aac81a676d52fd394f8e8c6a8612: Status 404 returned error can't find the container with id 0d176df2dd262c6d63ad22962fe79cc714b0aac81a676d52fd394f8e8c6a8612 Nov 02 18:15:53.303893 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:53.303876 5601 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:15:53.349284 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:53.349200 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" event={"ID":"851b2b14-e25a-47b9-8eab-0e68096e6268","Type":"ContainerStarted","Data":"0d176df2dd262c6d63ad22962fe79cc714b0aac81a676d52fd394f8e8c6a8612"} Nov 02 18:15:54.360438 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:54.360401 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" event={"ID":"851b2b14-e25a-47b9-8eab-0e68096e6268","Type":"ContainerStarted","Data":"79a03d67e4d2a8e53d948bf06497af0aebcb690ad72d041bfbc0ac55b26a9ce8"} Nov 02 18:15:54.360954 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:54.360470 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" Nov 02 18:15:54.382245 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:15:54.382196 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" podStartSLOduration=2.872299067 podStartE2EDuration="3.382182227s" totalImagesPullingTime="509.88316ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:54.378720481 +0000 UTC m=+2519.069774495" watchObservedRunningTime="2026-11-02 18:15:54.382182227 +0000 UTC m=+2519.073236240" Nov 02 18:16:00.376210 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.376166 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" Nov 02 18:16:00.469602 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.469557 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k"] Nov 02 18:16:00.479322 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.479154 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.488415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.488378 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k"] Nov 02 18:16:00.526498 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.526469 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0b0610-openshift-default-dockercfg-twq84\"" Nov 02 18:16:00.543335 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.543283 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tck28\" (UniqueName: \"kubernetes.io/projected/5c2aa306-9a80-4299-9ada-d7229e813dee-kube-api-access-tck28\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.543544 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.543367 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-token\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.543544 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.543452 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-data\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.543544 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.543506 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-podinfo\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.543790 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.543576 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-envoy\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.543790 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.543606 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-credential-socket\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.543790 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.543709 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-workload-socket\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.543790 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.543740 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-workload-certs\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.544015 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.543898 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2aa306-9a80-4299-9ada-d7229e813dee-istiod-ca-cert\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.659692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.654794 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2aa306-9a80-4299-9ada-d7229e813dee-istiod-ca-cert\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.659692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.654911 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tck28\" (UniqueName: \"kubernetes.io/projected/5c2aa306-9a80-4299-9ada-d7229e813dee-kube-api-access-tck28\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.659692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.654958 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-token\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.659692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.655039 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-data\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.659692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.655094 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-podinfo\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.659692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.655166 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-envoy\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.659692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.655204 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-credential-socket\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.659692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.655291 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-workload-socket\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.659692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.655321 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-workload-certs\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.659692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.655914 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-workload-certs\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.659692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.656595 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2aa306-9a80-4299-9ada-d7229e813dee-istiod-ca-cert\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.659692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.657364 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-data\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.659692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.657866 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-credential-socket\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.659692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.658390 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-workload-socket\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.728090 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.675150 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-envoy\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.728090 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.700218 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tck28\" (UniqueName: \"kubernetes.io/projected/5c2aa306-9a80-4299-9ada-d7229e813dee-kube-api-access-tck28\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.728090 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.700731 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-podinfo\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.728090 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.701288 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-token\") pod \"e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:00.808640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:00.808238 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:01.237965 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:01.237925 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k"] Nov 02 18:16:01.238280 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:16:01.238255 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2aa306_9a80_4299_9ada_d7229e813dee.slice/crio-462f9af4aa927b1d0fa7e42af1e096419635b06b1ffb3f54854a8b64470a4b4f WatchSource:0}: Error finding container 462f9af4aa927b1d0fa7e42af1e096419635b06b1ffb3f54854a8b64470a4b4f: Status 404 returned error can't find the container with id 462f9af4aa927b1d0fa7e42af1e096419635b06b1ffb3f54854a8b64470a4b4f Nov 02 18:16:01.242230 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:01.242159 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:16:01.242328 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:01.242251 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:16:01.242328 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:01.242306 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:16:01.473596 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:01.473541 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" event={"ID":"5c2aa306-9a80-4299-9ada-d7229e813dee","Type":"ContainerStarted","Data":"e931d5cf9949b1434fbe01a656b3bbc428c5257df06f89676106190f88818a29"} Nov 02 18:16:01.474105 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:01.473675 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" event={"ID":"5c2aa306-9a80-4299-9ada-d7229e813dee","Type":"ContainerStarted","Data":"462f9af4aa927b1d0fa7e42af1e096419635b06b1ffb3f54854a8b64470a4b4f"} Nov 02 18:16:01.513699 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:01.511184 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" podStartSLOduration=1.511164152 podStartE2EDuration="1.511164152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:01.50826404 +0000 UTC m=+2526.199318079" watchObservedRunningTime="2026-11-02 18:16:01.511164152 +0000 UTC m=+2526.202218165" Nov 02 18:16:01.809009 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:01.808903 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:03.269555 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:03.269512 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:03.588016 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:03.587938 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:03.588992 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:03.588955 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:16:04.811680 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:16:04.811635 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:16:06.084252 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:06.084215 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr"] Nov 02 18:16:06.090321 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:06.090297 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" Nov 02 18:16:06.098803 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:06.098775 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr"] Nov 02 18:16:06.099129 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:06.099093 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0b0610\"" Nov 02 18:16:06.100547 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:06.100523 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0b0610-dockercfg-v77zv\"" Nov 02 18:16:06.375691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:06.368490 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4fvw\" (UniqueName: \"kubernetes.io/projected/ae604043-587b-4fd4-b923-4bd1b6170df2-kube-api-access-g4fvw\") pod \"payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr\" (UID: \"ae604043-587b-4fd4-b923-4bd1b6170df2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" Nov 02 18:16:06.375691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:06.368711 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae604043-587b-4fd4-b923-4bd1b6170df2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr\" (UID: \"ae604043-587b-4fd4-b923-4bd1b6170df2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" Nov 02 18:16:06.673889 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:06.673012 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4fvw\" (UniqueName: \"kubernetes.io/projected/ae604043-587b-4fd4-b923-4bd1b6170df2-kube-api-access-g4fvw\") pod \"payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr\" (UID: \"ae604043-587b-4fd4-b923-4bd1b6170df2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" Nov 02 18:16:06.673889 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:06.673293 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae604043-587b-4fd4-b923-4bd1b6170df2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr\" (UID: \"ae604043-587b-4fd4-b923-4bd1b6170df2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" Nov 02 18:16:06.689798 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:06.685257 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae604043-587b-4fd4-b923-4bd1b6170df2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr\" (UID: \"ae604043-587b-4fd4-b923-4bd1b6170df2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" Nov 02 18:16:06.746182 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:06.746140 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4fvw\" (UniqueName: \"kubernetes.io/projected/ae604043-587b-4fd4-b923-4bd1b6170df2-kube-api-access-g4fvw\") pod \"payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr\" (UID: \"ae604043-587b-4fd4-b923-4bd1b6170df2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" Nov 02 18:16:06.960700 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:06.932536 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5"] Nov 02 18:16:07.034049 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:07.034003 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" Nov 02 18:16:07.674072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:07.674037 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr"] Nov 02 18:16:07.674442 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:16:07.674353 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae604043_587b_4fd4_b923_4bd1b6170df2.slice/crio-4285ce2309eecea08899bdca95e334e46f60be478132a43b56e131e054440e69 WatchSource:0}: Error finding container 4285ce2309eecea08899bdca95e334e46f60be478132a43b56e131e054440e69: Status 404 returned error can't find the container with id 4285ce2309eecea08899bdca95e334e46f60be478132a43b56e131e054440e69 Nov 02 18:16:07.768844 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:07.768809 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" event={"ID":"ae604043-587b-4fd4-b923-4bd1b6170df2","Type":"ContainerStarted","Data":"4285ce2309eecea08899bdca95e334e46f60be478132a43b56e131e054440e69"} Nov 02 18:16:08.777883 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:08.777842 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" event={"ID":"ae604043-587b-4fd4-b923-4bd1b6170df2","Type":"ContainerStarted","Data":"ff8e22864822f427997c81cbd655fc40f2a702a78d3be76a0d6759b4a868542a"} Nov 02 18:16:08.778379 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:08.777912 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" Nov 02 18:16:08.801562 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:08.801496 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" podStartSLOduration=2.801479539 podStartE2EDuration="2.801479539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:08.796896703 +0000 UTC m=+2533.487950716" watchObservedRunningTime="2026-11-02 18:16:08.801479539 +0000 UTC m=+2533.492533552" Nov 02 18:16:10.615153 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:10.615113 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm"] Nov 02 18:16:10.615670 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:10.615369 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" podUID="64eaca4e-6457-4551-8e3d-6188dc62b363" containerName="istio-proxy" containerID="cri-o://8e7bdc2ddd8fc45acc19641a5a4867ed9c2c85c81cb5cd4c8814fa3d8bc5b15f" gracePeriod=30 Nov 02 18:16:12.011609 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.011560 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" podUID="82593a64-c25f-4d0b-8e6b-78816f40984c" containerName="payload-processing" containerID="cri-o://956cab20f6cc90b13436973e479532b91a37ad0e56a1b5c45419ff93ae012f02" gracePeriod=25 Nov 02 18:16:12.511702 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.511675 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" Nov 02 18:16:12.610791 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.610758 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82593a64-c25f-4d0b-8e6b-78816f40984c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/82593a64-c25f-4d0b-8e6b-78816f40984c-provider-credentials\") pod \"82593a64-c25f-4d0b-8e6b-78816f40984c\" (UID: \"82593a64-c25f-4d0b-8e6b-78816f40984c\") " Nov 02 18:16:12.610978 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.610818 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82593a64-c25f-4d0b-8e6b-78816f40984c-kube-api-access-6wqjf\" (UniqueName: \"kubernetes.io/projected/82593a64-c25f-4d0b-8e6b-78816f40984c-kube-api-access-6wqjf\") pod \"82593a64-c25f-4d0b-8e6b-78816f40984c\" (UID: \"82593a64-c25f-4d0b-8e6b-78816f40984c\") " Nov 02 18:16:12.610978 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.610877 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82593a64-c25f-4d0b-8e6b-78816f40984c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82593a64-c25f-4d0b-8e6b-78816f40984c-routing-overlay-volume\") pod \"82593a64-c25f-4d0b-8e6b-78816f40984c\" (UID: \"82593a64-c25f-4d0b-8e6b-78816f40984c\") " Nov 02 18:16:12.611110 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.611085 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82593a64-c25f-4d0b-8e6b-78816f40984c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82593a64-c25f-4d0b-8e6b-78816f40984c-plugins-config-volume\") pod \"82593a64-c25f-4d0b-8e6b-78816f40984c\" (UID: \"82593a64-c25f-4d0b-8e6b-78816f40984c\") " Nov 02 18:16:12.611331 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.611306 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82593a64-c25f-4d0b-8e6b-78816f40984c-routing-overlay-volume" pod "82593a64-c25f-4d0b-8e6b-78816f40984c" (UID: "82593a64-c25f-4d0b-8e6b-78816f40984c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:16:12.611468 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.611444 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82593a64-c25f-4d0b-8e6b-78816f40984c-plugins-config-volume" pod "82593a64-c25f-4d0b-8e6b-78816f40984c" (UID: "82593a64-c25f-4d0b-8e6b-78816f40984c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:16:12.611984 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.611961 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82593a64-c25f-4d0b-8e6b-78816f40984c-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:12.612048 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.611993 5601 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82593a64-c25f-4d0b-8e6b-78816f40984c-routing-overlay-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:12.613227 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.613205 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82593a64-c25f-4d0b-8e6b-78816f40984c-kube-api-access-6wqjf" pod "82593a64-c25f-4d0b-8e6b-78816f40984c" (UID: "82593a64-c25f-4d0b-8e6b-78816f40984c"). InnerVolumeSpecName "kube-api-access-6wqjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:12.613339 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.613231 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82593a64-c25f-4d0b-8e6b-78816f40984c-provider-credentials" pod "82593a64-c25f-4d0b-8e6b-78816f40984c" (UID: "82593a64-c25f-4d0b-8e6b-78816f40984c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:12.714271 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.714236 5601 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/82593a64-c25f-4d0b-8e6b-78816f40984c-provider-credentials\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:12.714271 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.714270 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wqjf\" (UniqueName: \"kubernetes.io/projected/82593a64-c25f-4d0b-8e6b-78816f40984c-kube-api-access-6wqjf\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:12.841374 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.841337 5601 generic.go:356] "Generic (PLEG): container finished" podID="82593a64-c25f-4d0b-8e6b-78816f40984c" containerID="956cab20f6cc90b13436973e479532b91a37ad0e56a1b5c45419ff93ae012f02" exitCode=0 Nov 02 18:16:12.841580 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.841396 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" Nov 02 18:16:12.841580 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.841422 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" event={"ID":"82593a64-c25f-4d0b-8e6b-78816f40984c","Type":"ContainerDied","Data":"956cab20f6cc90b13436973e479532b91a37ad0e56a1b5c45419ff93ae012f02"} Nov 02 18:16:12.841580 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.841497 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5" event={"ID":"82593a64-c25f-4d0b-8e6b-78816f40984c","Type":"ContainerDied","Data":"aa658be7bd4b3738b30229904373e0be2b1a620baaf56f800484ad50de092c5a"} Nov 02 18:16:12.841580 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.841531 5601 scope.go:122] "RemoveContainer" containerID="956cab20f6cc90b13436973e479532b91a37ad0e56a1b5c45419ff93ae012f02" Nov 02 18:16:12.855390 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.855368 5601 scope.go:122] "RemoveContainer" containerID="956cab20f6cc90b13436973e479532b91a37ad0e56a1b5c45419ff93ae012f02" Nov 02 18:16:12.855744 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:16:12.855709 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"956cab20f6cc90b13436973e479532b91a37ad0e56a1b5c45419ff93ae012f02\": container with ID starting with 956cab20f6cc90b13436973e479532b91a37ad0e56a1b5c45419ff93ae012f02 not found: ID does not exist" containerID="956cab20f6cc90b13436973e479532b91a37ad0e56a1b5c45419ff93ae012f02" Nov 02 18:16:12.855872 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.855754 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956cab20f6cc90b13436973e479532b91a37ad0e56a1b5c45419ff93ae012f02"} err="failed to get container status \"956cab20f6cc90b13436973e479532b91a37ad0e56a1b5c45419ff93ae012f02\": rpc error: code = NotFound desc = could not find container \"956cab20f6cc90b13436973e479532b91a37ad0e56a1b5c45419ff93ae012f02\": container with ID starting with 956cab20f6cc90b13436973e479532b91a37ad0e56a1b5c45419ff93ae012f02 not found: ID does not exist" Nov 02 18:16:12.879358 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.879136 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5"] Nov 02 18:16:12.885269 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:12.885233 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-7dcc6jnf5"] Nov 02 18:16:13.442041 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:13.442008 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82593a64-c25f-4d0b-8e6b-78816f40984c" path="/var/lib/kubelet/pods/82593a64-c25f-4d0b-8e6b-78816f40984c/volumes" Nov 02 18:16:15.874768 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:15.874724 5601 generic.go:356] "Generic (PLEG): container finished" podID="64eaca4e-6457-4551-8e3d-6188dc62b363" containerID="8e7bdc2ddd8fc45acc19641a5a4867ed9c2c85c81cb5cd4c8814fa3d8bc5b15f" exitCode=0 Nov 02 18:16:15.875276 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:15.874775 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" event={"ID":"64eaca4e-6457-4551-8e3d-6188dc62b363","Type":"ContainerDied","Data":"8e7bdc2ddd8fc45acc19641a5a4867ed9c2c85c81cb5cd4c8814fa3d8bc5b15f"} Nov 02 18:16:15.986318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:15.985111 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 18:16:16.114062 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.114028 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-envoy\") pod \"64eaca4e-6457-4551-8e3d-6188dc62b363\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " Nov 02 18:16:16.114284 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.114082 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64eaca4e-6457-4551-8e3d-6188dc62b363-kube-api-access-pvl8x\" (UniqueName: \"kubernetes.io/projected/64eaca4e-6457-4551-8e3d-6188dc62b363-kube-api-access-pvl8x\") pod \"64eaca4e-6457-4551-8e3d-6188dc62b363\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " Nov 02 18:16:16.114284 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.114161 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-workload-socket\") pod \"64eaca4e-6457-4551-8e3d-6188dc62b363\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " Nov 02 18:16:16.114422 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.114286 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-credential-socket\") pod \"64eaca4e-6457-4551-8e3d-6188dc62b363\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " Nov 02 18:16:16.114422 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.114350 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-podinfo\") pod \"64eaca4e-6457-4551-8e3d-6188dc62b363\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " Nov 02 18:16:16.114540 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.114451 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-token\" (UniqueName: \"kubernetes.io/projected/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-token\") pod \"64eaca4e-6457-4551-8e3d-6188dc62b363\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " Nov 02 18:16:16.114595 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.114573 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-workload-socket" pod "64eaca4e-6457-4551-8e3d-6188dc62b363" (UID: "64eaca4e-6457-4551-8e3d-6188dc62b363"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:16.114673 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.114592 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-data\") pod \"64eaca4e-6457-4551-8e3d-6188dc62b363\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " Nov 02 18:16:16.114744 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.114715 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-workload-certs\") pod \"64eaca4e-6457-4551-8e3d-6188dc62b363\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " Nov 02 18:16:16.114802 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.114786 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64eaca4e-6457-4551-8e3d-6188dc62b363-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64eaca4e-6457-4551-8e3d-6188dc62b363-istiod-ca-cert\") pod \"64eaca4e-6457-4551-8e3d-6188dc62b363\" (UID: \"64eaca4e-6457-4551-8e3d-6188dc62b363\") " Nov 02 18:16:16.115780 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.115141 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-data" pod "64eaca4e-6457-4551-8e3d-6188dc62b363" (UID: "64eaca4e-6457-4551-8e3d-6188dc62b363"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:16.115780 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.115171 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-workload-certs" pod "64eaca4e-6457-4551-8e3d-6188dc62b363" (UID: "64eaca4e-6457-4551-8e3d-6188dc62b363"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:16.115780 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.115393 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-credential-socket" pod "64eaca4e-6457-4551-8e3d-6188dc62b363" (UID: "64eaca4e-6457-4551-8e3d-6188dc62b363"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:16.115780 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.115437 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64eaca4e-6457-4551-8e3d-6188dc62b363-istiod-ca-cert" pod "64eaca4e-6457-4551-8e3d-6188dc62b363" (UID: "64eaca4e-6457-4551-8e3d-6188dc62b363"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:16:16.115780 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.115663 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:16.115780 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.115687 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:16.115780 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.115702 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64eaca4e-6457-4551-8e3d-6188dc62b363-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:16.115780 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.115717 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:16.115780 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.115730 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:16.116933 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.116907 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-envoy" pod "64eaca4e-6457-4551-8e3d-6188dc62b363" (UID: "64eaca4e-6457-4551-8e3d-6188dc62b363"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:16.117729 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.117693 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64eaca4e-6457-4551-8e3d-6188dc62b363-kube-api-access-pvl8x" pod "64eaca4e-6457-4551-8e3d-6188dc62b363" (UID: "64eaca4e-6457-4551-8e3d-6188dc62b363"). InnerVolumeSpecName "kube-api-access-pvl8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:16.117854 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.117730 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-token" pod "64eaca4e-6457-4551-8e3d-6188dc62b363" (UID: "64eaca4e-6457-4551-8e3d-6188dc62b363"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:16.118262 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.118239 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-podinfo" pod "64eaca4e-6457-4551-8e3d-6188dc62b363" (UID: "64eaca4e-6457-4551-8e3d-6188dc62b363"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:16:16.217447 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.217413 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:16.217447 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.217452 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:16.217712 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.217462 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64eaca4e-6457-4551-8e3d-6188dc62b363-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:16.217712 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.217472 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvl8x\" (UniqueName: \"kubernetes.io/projected/64eaca4e-6457-4551-8e3d-6188dc62b363-kube-api-access-pvl8x\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:16.953070 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.953029 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" event={"ID":"64eaca4e-6457-4551-8e3d-6188dc62b363","Type":"ContainerDied","Data":"a8735216eb50873b97e2575f2a4d03baaa91a3f15425d433cd95e99c54179d87"} Nov 02 18:16:16.953509 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.953102 5601 scope.go:122] "RemoveContainer" containerID="8e7bdc2ddd8fc45acc19641a5a4867ed9c2c85c81cb5cd4c8814fa3d8bc5b15f" Nov 02 18:16:16.953509 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.953287 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm" Nov 02 18:16:16.987351 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.987314 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm"] Nov 02 18:16:16.991515 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:16.991468 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-r86lm"] Nov 02 18:16:17.429765 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:17.429724 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64eaca4e-6457-4551-8e3d-6188dc62b363" path="/var/lib/kubelet/pods/64eaca4e-6457-4551-8e3d-6188dc62b363/volumes" Nov 02 18:16:19.818094 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:19.818060 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" Nov 02 18:16:21.589773 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:21.589714 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 18:16:21.607719 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:21.607671 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:21.941990 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:21.940740 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:22.129643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:22.123576 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:22.147653 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:22.144805 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:22.230709 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:22.229718 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:22.451640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:22.450719 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:22.608697 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:22.607885 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:22.608697 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:22.608492 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 18:16:22.625643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:22.613237 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:22.650638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:22.637752 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 18:16:22.691640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:22.691550 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:22.692644 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:22.691960 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 18:16:22.748373 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:22.747658 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:22.952650 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:22.946976 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 02 18:16:22.952650 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:22.947453 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 18:16:23.215696 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:23.165411 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 02 18:16:23.217647 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:23.216642 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 18:16:23.344056 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:23.331760 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:23.346602 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:23.345438 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:23.346602 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:23.346269 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 18:16:23.618560 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:23.618449 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:23.624234 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:23.624013 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:23.694809 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:23.694756 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 18:16:23.704095 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:23.704055 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 18:16:23.749212 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:23.748075 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:23.898014 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:23.876105 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:23.921838 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:23.921802 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 18:16:23.922121 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:23.922104 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 18:16:23.922441 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:23.922422 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 18:16:23.922610 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:23.922595 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 18:16:23.948649 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:23.944048 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 18:16:23.958649 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:23.958390 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 18:16:24.053931 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:24.053886 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:24.076756 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:24.065261 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 18:16:24.110434 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:24.110406 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 18:16:24.110756 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:24.110740 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 18:16:24.309784 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:24.309736 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:24.367070 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:24.367026 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 18:16:24.514465 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:24.514432 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 18:16:24.519806 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:24.519228 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 18:16:29.788170 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:29.787772 5601 patch_prober.go:29] interesting pod/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:16:29.788170 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:29.787848 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" podUID="945a032d-c69f-4d9e-96a5-db1ac483af67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:31.104806 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:31.104766 5601 patch_prober.go:29] interesting pod/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:16:31.105237 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:31.104839 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" podUID="208dcda2-8e4e-4f47-af69-942e7412eda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:16:33.002653 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:33.002542 5601 generic.go:356] "Generic (PLEG): container finished" podID="208dcda2-8e4e-4f47-af69-942e7412eda9" containerID="b946e58b3b2011581fc25302f71f1f739853e738a3c262b9eb7833e3307b6d19" exitCode=0 Nov 02 18:16:33.002653 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:33.002626 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" event={"ID":"208dcda2-8e4e-4f47-af69-942e7412eda9","Type":"ContainerDied","Data":"b946e58b3b2011581fc25302f71f1f739853e738a3c262b9eb7833e3307b6d19"} Nov 02 18:16:33.003173 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:33.003102 5601 scope.go:122] "RemoveContainer" containerID="b946e58b3b2011581fc25302f71f1f739853e738a3c262b9eb7833e3307b6d19" Nov 02 18:16:33.005240 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:33.004735 5601 generic.go:356] "Generic (PLEG): container finished" podID="945a032d-c69f-4d9e-96a5-db1ac483af67" containerID="4f94841406dd48759f969b3f451b0570036eb289bfe818515e0b5e26a9451b3d" exitCode=0 Nov 02 18:16:33.005240 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:33.004792 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" event={"ID":"945a032d-c69f-4d9e-96a5-db1ac483af67","Type":"ContainerDied","Data":"4f94841406dd48759f969b3f451b0570036eb289bfe818515e0b5e26a9451b3d"} Nov 02 18:16:33.005240 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:33.005207 5601 scope.go:122] "RemoveContainer" containerID="4f94841406dd48759f969b3f451b0570036eb289bfe818515e0b5e26a9451b3d" Nov 02 18:16:33.005700 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:33.005663 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:16:33.005856 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:33.005754 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:16:33.005856 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:33.005799 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:16:33.007424 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:33.007393 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:16:33.007539 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:33.007504 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:16:33.007607 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:33.007559 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:16:34.022661 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:34.022585 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" event={"ID":"945a032d-c69f-4d9e-96a5-db1ac483af67","Type":"ContainerStarted","Data":"61b59d47ca148c52949dc9e30caf2d91fe8cbdfd630f3e1e387b26f89a060648"} Nov 02 18:16:34.025248 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:34.025221 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" event={"ID":"208dcda2-8e4e-4f47-af69-942e7412eda9","Type":"ContainerStarted","Data":"ae76333d5fbb976dc13fb348e995009d0009545eceabcbf5d19515198fcee852"} Nov 02 18:16:34.514324 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:16:34.514285 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:16:34.722213 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:34.722171 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:16:34.722398 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:34.722362 5601 patch_prober.go:29] interesting pod/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 02 18:16:34.722444 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:34.722422 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" podUID="945a032d-c69f-4d9e-96a5-db1ac483af67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 02 18:16:34.876695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:34.876452 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:16:34.876695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:34.876546 5601 patch_prober.go:29] interesting pod/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 02 18:16:34.876944 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:34.876730 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" podUID="208dcda2-8e4e-4f47-af69-942e7412eda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 02 18:16:35.721967 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:35.721931 5601 patch_prober.go:29] interesting pod/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 02 18:16:35.722445 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:35.721990 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" podUID="945a032d-c69f-4d9e-96a5-db1ac483af67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 02 18:16:35.882715 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:35.882039 5601 patch_prober.go:29] interesting pod/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 02 18:16:35.882715 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:35.882121 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" podUID="208dcda2-8e4e-4f47-af69-942e7412eda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 02 18:16:36.402847 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.402133 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg"] Nov 02 18:16:36.404515 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.404481 5601 cpu_manager.go:414] "containerMap: removing container" podUID="64eaca4e-6457-4551-8e3d-6188dc62b363" containerName="istio-proxy" Nov 02 18:16:36.404718 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.404701 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64eaca4e-6457-4551-8e3d-6188dc62b363" containerName="istio-proxy" Nov 02 18:16:36.404862 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.404851 5601 cpu_manager.go:414] "containerMap: removing container" podUID="82593a64-c25f-4d0b-8e6b-78816f40984c" containerName="payload-processing" Nov 02 18:16:36.404962 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.404951 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82593a64-c25f-4d0b-8e6b-78816f40984c" containerName="payload-processing" Nov 02 18:16:36.405513 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.405377 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="64eaca4e-6457-4551-8e3d-6188dc62b363" containerName="istio-proxy" Nov 02 18:16:36.405687 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.405670 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="82593a64-c25f-4d0b-8e6b-78816f40984c" containerName="payload-processing" Nov 02 18:16:36.411546 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.409440 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg"] Nov 02 18:16:36.411546 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.409593 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" Nov 02 18:16:36.414538 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.414511 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0b0610-serving-cert\"" Nov 02 18:16:36.630335 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.630290 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5717d304-fd02-42a4-a04f-0958ce43fdb0-metrics-certs\") pod \"maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg\" (UID: \"5717d304-fd02-42a4-a04f-0958ce43fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" Nov 02 18:16:36.630542 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.630374 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56fwp\" (UniqueName: \"kubernetes.io/projected/5717d304-fd02-42a4-a04f-0958ce43fdb0-kube-api-access-56fwp\") pod \"maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg\" (UID: \"5717d304-fd02-42a4-a04f-0958ce43fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" Nov 02 18:16:36.630542 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.630425 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5717d304-fd02-42a4-a04f-0958ce43fdb0-maas-api-tls\") pod \"maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg\" (UID: \"5717d304-fd02-42a4-a04f-0958ce43fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" Nov 02 18:16:36.630542 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.630462 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5717d304-fd02-42a4-a04f-0958ce43fdb0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg\" (UID: \"5717d304-fd02-42a4-a04f-0958ce43fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" Nov 02 18:16:36.721786 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.721753 5601 patch_prober.go:29] interesting pod/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 02 18:16:36.721964 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.721806 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" podUID="945a032d-c69f-4d9e-96a5-db1ac483af67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 02 18:16:36.733201 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.733167 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5717d304-fd02-42a4-a04f-0958ce43fdb0-metrics-certs\") pod \"maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg\" (UID: \"5717d304-fd02-42a4-a04f-0958ce43fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" Nov 02 18:16:36.733584 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.733288 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56fwp\" (UniqueName: \"kubernetes.io/projected/5717d304-fd02-42a4-a04f-0958ce43fdb0-kube-api-access-56fwp\") pod \"maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg\" (UID: \"5717d304-fd02-42a4-a04f-0958ce43fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" Nov 02 18:16:36.733584 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.733379 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5717d304-fd02-42a4-a04f-0958ce43fdb0-maas-api-tls\") pod \"maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg\" (UID: \"5717d304-fd02-42a4-a04f-0958ce43fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" Nov 02 18:16:36.733584 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.733457 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5717d304-fd02-42a4-a04f-0958ce43fdb0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg\" (UID: \"5717d304-fd02-42a4-a04f-0958ce43fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" Nov 02 18:16:36.748646 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.734240 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5717d304-fd02-42a4-a04f-0958ce43fdb0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg\" (UID: \"5717d304-fd02-42a4-a04f-0958ce43fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" Nov 02 18:16:36.766386 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.766352 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56fwp\" (UniqueName: \"kubernetes.io/projected/5717d304-fd02-42a4-a04f-0958ce43fdb0-kube-api-access-56fwp\") pod \"maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg\" (UID: \"5717d304-fd02-42a4-a04f-0958ce43fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" Nov 02 18:16:36.782215 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.781660 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5717d304-fd02-42a4-a04f-0958ce43fdb0-maas-api-tls\") pod \"maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg\" (UID: \"5717d304-fd02-42a4-a04f-0958ce43fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" Nov 02 18:16:36.783766 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.783573 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5717d304-fd02-42a4-a04f-0958ce43fdb0-metrics-certs\") pod \"maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg\" (UID: \"5717d304-fd02-42a4-a04f-0958ce43fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" Nov 02 18:16:36.895311 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.895256 5601 patch_prober.go:29] interesting pod/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 02 18:16:36.895510 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:36.895323 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" podUID="208dcda2-8e4e-4f47-af69-942e7412eda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 02 18:16:37.040774 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:37.033944 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" Nov 02 18:16:37.694177 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:37.694144 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg"] Nov 02 18:16:37.694925 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:16:37.694889 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5717d304_fd02_42a4_a04f_0958ce43fdb0.slice/crio-c929077793035509f326b6934d1e67f2a95a6e33310bde3703fa461028d3f7ca WatchSource:0}: Error finding container c929077793035509f326b6934d1e67f2a95a6e33310bde3703fa461028d3f7ca: Status 404 returned error can't find the container with id c929077793035509f326b6934d1e67f2a95a6e33310bde3703fa461028d3f7ca Nov 02 18:16:37.728029 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:37.727490 5601 patch_prober.go:29] interesting pod/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 02 18:16:37.728029 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:37.727551 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" podUID="945a032d-c69f-4d9e-96a5-db1ac483af67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 02 18:16:37.884639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:37.884100 5601 patch_prober.go:29] interesting pod/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 02 18:16:37.884639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:37.884160 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" podUID="208dcda2-8e4e-4f47-af69-942e7412eda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 02 18:16:38.200763 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:38.200723 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" event={"ID":"5717d304-fd02-42a4-a04f-0958ce43fdb0","Type":"ContainerStarted","Data":"c929077793035509f326b6934d1e67f2a95a6e33310bde3703fa461028d3f7ca"} Nov 02 18:16:38.721570 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:38.721533 5601 patch_prober.go:29] interesting pod/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 02 18:16:38.721783 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:38.721595 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" podUID="945a032d-c69f-4d9e-96a5-db1ac483af67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 02 18:16:38.884545 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:38.884073 5601 patch_prober.go:29] interesting pod/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 02 18:16:38.884545 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:38.884135 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" podUID="208dcda2-8e4e-4f47-af69-942e7412eda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 02 18:16:39.211157 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:39.211114 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" event={"ID":"5717d304-fd02-42a4-a04f-0958ce43fdb0","Type":"ContainerStarted","Data":"bac52e3a96202835bce66d2579634249e819254f62177d70d266894153095460"} Nov 02 18:16:39.211416 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:39.211200 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" Nov 02 18:16:39.231447 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:39.231399 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" podStartSLOduration=2.6344076149999998 podStartE2EDuration="3.231385836s" totalImagesPullingTime="596.978221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:39.229057803 +0000 UTC m=+2563.920111839" watchObservedRunningTime="2026-11-02 18:16:39.231385836 +0000 UTC m=+2563.922439847" Nov 02 18:16:39.721778 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:39.721740 5601 patch_prober.go:29] interesting pod/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 02 18:16:39.721964 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:39.721796 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" podUID="945a032d-c69f-4d9e-96a5-db1ac483af67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 02 18:16:39.883641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:39.883116 5601 patch_prober.go:29] interesting pod/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 02 18:16:39.883641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:39.883188 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" podUID="208dcda2-8e4e-4f47-af69-942e7412eda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 02 18:16:40.727651 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:40.727588 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:16:40.728020 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:40.727871 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:16:40.728566 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:40.728546 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:16:40.880837 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:40.880806 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:16:40.881340 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:40.881321 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:16:40.882091 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:40.882069 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:16:45.225188 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:45.225154 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" Nov 02 18:16:56.711794 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:56.711756 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf"] Nov 02 18:16:56.716741 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:56.716695 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" Nov 02 18:16:56.726409 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:56.726378 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-63be7a-serving-cert\"" Nov 02 18:16:56.734723 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:56.734693 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf"] Nov 02 18:16:56.781868 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:56.781820 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5be85144-c6c7-4399-9837-737b76d4e114-metrics-certs\") pod \"maas-api-e2e-inf-b-63be7a-79b594b699-px7cf\" (UID: \"5be85144-c6c7-4399-9837-737b76d4e114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" Nov 02 18:16:56.782060 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:56.781943 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be85144-c6c7-4399-9837-737b76d4e114-openshift-service-ca\") pod \"maas-api-e2e-inf-b-63be7a-79b594b699-px7cf\" (UID: \"5be85144-c6c7-4399-9837-737b76d4e114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" Nov 02 18:16:56.782060 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:56.781982 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbmz8\" (UniqueName: \"kubernetes.io/projected/5be85144-c6c7-4399-9837-737b76d4e114-kube-api-access-kbmz8\") pod \"maas-api-e2e-inf-b-63be7a-79b594b699-px7cf\" (UID: \"5be85144-c6c7-4399-9837-737b76d4e114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" Nov 02 18:16:56.782060 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:56.782033 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5be85144-c6c7-4399-9837-737b76d4e114-maas-api-tls\") pod \"maas-api-e2e-inf-b-63be7a-79b594b699-px7cf\" (UID: \"5be85144-c6c7-4399-9837-737b76d4e114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" Nov 02 18:16:56.884353 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:56.884163 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be85144-c6c7-4399-9837-737b76d4e114-openshift-service-ca\") pod \"maas-api-e2e-inf-b-63be7a-79b594b699-px7cf\" (UID: \"5be85144-c6c7-4399-9837-737b76d4e114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" Nov 02 18:16:56.884353 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:56.884264 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbmz8\" (UniqueName: \"kubernetes.io/projected/5be85144-c6c7-4399-9837-737b76d4e114-kube-api-access-kbmz8\") pod \"maas-api-e2e-inf-b-63be7a-79b594b699-px7cf\" (UID: \"5be85144-c6c7-4399-9837-737b76d4e114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" Nov 02 18:16:56.884353 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:56.884352 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5be85144-c6c7-4399-9837-737b76d4e114-maas-api-tls\") pod \"maas-api-e2e-inf-b-63be7a-79b594b699-px7cf\" (UID: \"5be85144-c6c7-4399-9837-737b76d4e114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" Nov 02 18:16:56.884695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:56.884602 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5be85144-c6c7-4399-9837-737b76d4e114-metrics-certs\") pod \"maas-api-e2e-inf-b-63be7a-79b594b699-px7cf\" (UID: \"5be85144-c6c7-4399-9837-737b76d4e114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" Nov 02 18:16:56.890166 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:56.887104 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be85144-c6c7-4399-9837-737b76d4e114-openshift-service-ca\") pod \"maas-api-e2e-inf-b-63be7a-79b594b699-px7cf\" (UID: \"5be85144-c6c7-4399-9837-737b76d4e114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" Nov 02 18:16:56.894970 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:56.894917 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5be85144-c6c7-4399-9837-737b76d4e114-metrics-certs\") pod \"maas-api-e2e-inf-b-63be7a-79b594b699-px7cf\" (UID: \"5be85144-c6c7-4399-9837-737b76d4e114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" Nov 02 18:16:56.897379 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:56.896945 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbmz8\" (UniqueName: \"kubernetes.io/projected/5be85144-c6c7-4399-9837-737b76d4e114-kube-api-access-kbmz8\") pod \"maas-api-e2e-inf-b-63be7a-79b594b699-px7cf\" (UID: \"5be85144-c6c7-4399-9837-737b76d4e114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" Nov 02 18:16:56.897379 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:56.897056 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5be85144-c6c7-4399-9837-737b76d4e114-maas-api-tls\") pod \"maas-api-e2e-inf-b-63be7a-79b594b699-px7cf\" (UID: \"5be85144-c6c7-4399-9837-737b76d4e114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" Nov 02 18:16:57.038401 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:57.038311 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" Nov 02 18:16:57.462231 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:57.462197 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf"] Nov 02 18:16:58.406718 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:58.406685 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" event={"ID":"5be85144-c6c7-4399-9837-737b76d4e114","Type":"ContainerStarted","Data":"6e5d55db4f5ff09a020f2777d2d21c044f5cd75e1e68cab8e587b60800246552"} Nov 02 18:16:58.407156 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:58.406742 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" event={"ID":"5be85144-c6c7-4399-9837-737b76d4e114","Type":"ContainerStarted","Data":"78806d11660f9dcf0378ab5a7b3818e3c6913e41beec1318e1c5d17e6d2a8799"} Nov 02 18:16:58.407156 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:58.406768 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" Nov 02 18:16:58.429153 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:16:58.429105 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" podStartSLOduration=2.022119568 podStartE2EDuration="2.429091676s" totalImagesPullingTime="406.972108ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:58.427366743 +0000 UTC m=+2583.118420755" watchObservedRunningTime="2026-11-02 18:16:58.429091676 +0000 UTC m=+2583.120145688" Nov 02 18:17:04.419284 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:04.419254 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" Nov 02 18:17:04.538097 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:17:04.538052 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:17:19.911440 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:19.911397 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj"] Nov 02 18:17:19.917099 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:19.917072 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:19.926072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:19.926041 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj"] Nov 02 18:17:19.951378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:19.951353 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-06644b-openshift-default-dockercfg-hvpnl\"" Nov 02 18:17:20.072944 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.072893 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-data\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.090769 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.090725 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-envoy\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.101013 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.100974 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-workload-socket\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.127763 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.127725 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-token\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.128104 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.128078 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istiod-ca-cert\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.128493 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.128469 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6qsr\" (UniqueName: \"kubernetes.io/projected/ea7a3828-ae1f-4f34-a847-3a03b824cd33-kube-api-access-t6qsr\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.129062 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.129036 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-workload-certs\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.129301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.129283 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-credential-socket\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.129480 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.129464 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-podinfo\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.248639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.235664 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-data\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.248639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.235733 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-envoy\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.248639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.235847 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-workload-socket\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.248639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.235883 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-token\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.248639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.235914 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istiod-ca-cert\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.248639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.236090 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6qsr\" (UniqueName: \"kubernetes.io/projected/ea7a3828-ae1f-4f34-a847-3a03b824cd33-kube-api-access-t6qsr\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.248639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.236168 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-workload-certs\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.248639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.236248 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-credential-socket\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.248639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.236281 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-podinfo\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.248639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.244377 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-data\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.250501 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.250469 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-workload-socket\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.270701 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.255160 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-workload-certs\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.271935 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.271594 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istiod-ca-cert\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.271935 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.271874 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-credential-socket\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.273152 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.273127 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-envoy\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.293129 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.293095 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-token\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.293940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.293878 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-podinfo\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.303640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.300926 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6qsr\" (UniqueName: \"kubernetes.io/projected/ea7a3828-ae1f-4f34-a847-3a03b824cd33-kube-api-access-t6qsr\") pod \"e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.539393 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.539289 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:20.847372 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.847340 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj"] Nov 02 18:17:20.848168 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:17:20.848137 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7a3828_ae1f_4f34_a847_3a03b824cd33.slice/crio-46af90cc9bf9d712abc6965328810fbc5db4f83a7f40988d3f6d174577cd458a WatchSource:0}: Error finding container 46af90cc9bf9d712abc6965328810fbc5db4f83a7f40988d3f6d174577cd458a: Status 404 returned error can't find the container with id 46af90cc9bf9d712abc6965328810fbc5db4f83a7f40988d3f6d174577cd458a Nov 02 18:17:20.852694 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.852651 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:17:20.852793 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.852730 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:17:20.852793 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.852762 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:17:20.920349 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:20.920306 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" event={"ID":"ea7a3828-ae1f-4f34-a847-3a03b824cd33","Type":"ContainerStarted","Data":"46af90cc9bf9d712abc6965328810fbc5db4f83a7f40988d3f6d174577cd458a"} Nov 02 18:17:21.931286 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:21.931252 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" event={"ID":"ea7a3828-ae1f-4f34-a847-3a03b824cd33","Type":"ContainerStarted","Data":"a047143d12c6314cf0662fbff8e7c329aced046d5435f2e7e0c0bb62585ea600"} Nov 02 18:17:21.953552 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:21.953494 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" podStartSLOduration=2.953439731 podStartE2EDuration="2.953439731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:21.952822163 +0000 UTC m=+2606.643876176" watchObservedRunningTime="2026-11-02 18:17:21.953439731 +0000 UTC m=+2606.644493807" Nov 02 18:17:22.539785 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:22.539750 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:22.546505 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:22.546479 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:22.939249 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:22.939220 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:22.940181 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:22.940159 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:17:25.594898 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:25.594855 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7"] Nov 02 18:17:25.599822 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:25.599797 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" Nov 02 18:17:25.606466 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:25.606427 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-06644b\"" Nov 02 18:17:25.607392 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:25.607371 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-06644b-dockercfg-g27js\"" Nov 02 18:17:25.609566 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:25.609540 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7"] Nov 02 18:17:25.614775 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:25.614746 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7\" (UID: \"6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" Nov 02 18:17:25.614939 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:25.614921 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pflb\" (UniqueName: \"kubernetes.io/projected/6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed-kube-api-access-8pflb\") pod \"payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7\" (UID: \"6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" Nov 02 18:17:25.635050 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:25.635020 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw"] Nov 02 18:17:25.642543 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:25.642514 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw" Nov 02 18:17:25.645904 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:25.645870 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw"] Nov 02 18:17:25.789639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:25.730372 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7\" (UID: \"6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" Nov 02 18:17:25.789639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:25.730514 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pflb\" (UniqueName: \"kubernetes.io/projected/6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed-kube-api-access-8pflb\") pod \"payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7\" (UID: \"6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" Nov 02 18:17:25.871697 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:25.848785 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f29635-35c4-419c-80e4-0239daa2b244-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw\" (UID: \"c9f29635-35c4-419c-80e4-0239daa2b244\") " pod="openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw" Nov 02 18:17:25.872486 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:25.872448 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n44s9\" (UniqueName: \"kubernetes.io/projected/c9f29635-35c4-419c-80e4-0239daa2b244-kube-api-access-n44s9\") pod \"payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw\" (UID: \"c9f29635-35c4-419c-80e4-0239daa2b244\") " pod="openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw" Nov 02 18:17:25.873521 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:25.873498 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7\" (UID: \"6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" Nov 02 18:17:26.029712 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:25.993678 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f29635-35c4-419c-80e4-0239daa2b244-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw\" (UID: \"c9f29635-35c4-419c-80e4-0239daa2b244\") " pod="openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw" Nov 02 18:17:26.029712 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:26.019325 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f29635-35c4-419c-80e4-0239daa2b244-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw\" (UID: \"c9f29635-35c4-419c-80e4-0239daa2b244\") " pod="openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw" Nov 02 18:17:26.029712 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:26.020945 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pflb\" (UniqueName: \"kubernetes.io/projected/6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed-kube-api-access-8pflb\") pod \"payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7\" (UID: \"6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" Nov 02 18:17:26.046640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:26.030234 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n44s9\" (UniqueName: \"kubernetes.io/projected/c9f29635-35c4-419c-80e4-0239daa2b244-kube-api-access-n44s9\") pod \"payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw\" (UID: \"c9f29635-35c4-419c-80e4-0239daa2b244\") " pod="openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw" Nov 02 18:17:26.046640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:26.039681 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n44s9\" (UniqueName: \"kubernetes.io/projected/c9f29635-35c4-419c-80e4-0239daa2b244-kube-api-access-n44s9\") pod \"payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw\" (UID: \"c9f29635-35c4-419c-80e4-0239daa2b244\") " pod="openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw" Nov 02 18:17:26.218809 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:26.218778 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" Nov 02 18:17:26.262208 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:26.262173 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw" Nov 02 18:17:26.513725 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:26.513693 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7"] Nov 02 18:17:26.516063 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:17:26.516029 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6273dace_6c1a_4bb9_bcd6_3c4e0fbc62ed.slice/crio-63230317c4100296e202b9822ebd61de595ba274462d61c54ff149fb4b511b23 WatchSource:0}: Error finding container 63230317c4100296e202b9822ebd61de595ba274462d61c54ff149fb4b511b23: Status 404 returned error can't find the container with id 63230317c4100296e202b9822ebd61de595ba274462d61c54ff149fb4b511b23 Nov 02 18:17:26.632684 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:26.632651 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw"] Nov 02 18:17:26.633106 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:17:26.632971 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f29635_35c4_419c_80e4_0239daa2b244.slice/crio-8f7c0e3b40d12c773165ac24e08e9c8cb029ee382041f915dfc3f384ea5b6796 WatchSource:0}: Error finding container 8f7c0e3b40d12c773165ac24e08e9c8cb029ee382041f915dfc3f384ea5b6796: Status 404 returned error can't find the container with id 8f7c0e3b40d12c773165ac24e08e9c8cb029ee382041f915dfc3f384ea5b6796 Nov 02 18:17:27.047505 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:27.047471 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw" event={"ID":"c9f29635-35c4-419c-80e4-0239daa2b244","Type":"ContainerStarted","Data":"8e5879bd867f0f4c36aadedb3e27b28d311875648da3fcea03ab1e0ffd0d2e10"} Nov 02 18:17:27.047762 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:27.047532 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw" event={"ID":"c9f29635-35c4-419c-80e4-0239daa2b244","Type":"ContainerStarted","Data":"8f7c0e3b40d12c773165ac24e08e9c8cb029ee382041f915dfc3f384ea5b6796"} Nov 02 18:17:27.047762 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:27.047563 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw" Nov 02 18:17:27.049293 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:27.049262 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" event={"ID":"6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed","Type":"ContainerStarted","Data":"e890b62ac95a337fb4cfb9bb42400c0ce1f6136ab92587ac07aa67e3c12f49a2"} Nov 02 18:17:27.049378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:27.049310 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" event={"ID":"6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed","Type":"ContainerStarted","Data":"63230317c4100296e202b9822ebd61de595ba274462d61c54ff149fb4b511b23"} Nov 02 18:17:27.049378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:27.049333 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" Nov 02 18:17:27.071301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:27.071258 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw" podStartSLOduration=2.071246447 podStartE2EDuration="2.071246447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:27.065602869 +0000 UTC m=+2611.756656880" watchObservedRunningTime="2026-11-02 18:17:27.071246447 +0000 UTC m=+2611.762300458" Nov 02 18:17:27.087294 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:27.087253 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" podStartSLOduration=2.08724111 podStartE2EDuration="2.08724111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:27.086769072 +0000 UTC m=+2611.777823083" watchObservedRunningTime="2026-11-02 18:17:27.08724111 +0000 UTC m=+2611.778295121" Nov 02 18:17:34.486340 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:17:34.486293 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:17:38.060373 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:38.060149 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" Nov 02 18:17:38.060793 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:38.060447 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw" Nov 02 18:17:40.619004 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.618958 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz"] Nov 02 18:17:40.626582 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.626548 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:40.638245 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.637313 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-63be7a\"/\"default-dockercfg-4db27\"" Nov 02 18:17:40.638245 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.637440 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-63be7a\"/\"kube-root-ca.crt\"" Nov 02 18:17:40.638245 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.637515 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-63be7a\"/\"openshift-service-ca.crt\"" Nov 02 18:17:40.638245 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.637682 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-63be7a\"/\"test-model-63be7a-kserve-self-signed-certs\"" Nov 02 18:17:40.644805 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.644761 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz"] Nov 02 18:17:40.666305 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.666258 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-dshm\") pod \"test-model-63be7a-kserve-7bbf984f67-sp5cz\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:40.666431 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.666418 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-tmp-dir\") pod \"test-model-63be7a-kserve-7bbf984f67-sp5cz\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:40.666537 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.666521 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-home\") pod \"test-model-63be7a-kserve-7bbf984f67-sp5cz\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:40.666576 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.666560 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2db6\" (UniqueName: \"kubernetes.io/projected/ab48f110-7cc1-45fb-a416-527d9c919917-kube-api-access-n2db6\") pod \"test-model-63be7a-kserve-7bbf984f67-sp5cz\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:40.666668 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.666655 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab48f110-7cc1-45fb-a416-527d9c919917-tls-certs\") pod \"test-model-63be7a-kserve-7bbf984f67-sp5cz\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:40.666770 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.666752 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-model-cache\") pod \"test-model-63be7a-kserve-7bbf984f67-sp5cz\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:40.770356 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.768703 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-home\") pod \"test-model-63be7a-kserve-7bbf984f67-sp5cz\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:40.770356 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.768784 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2db6\" (UniqueName: \"kubernetes.io/projected/ab48f110-7cc1-45fb-a416-527d9c919917-kube-api-access-n2db6\") pod \"test-model-63be7a-kserve-7bbf984f67-sp5cz\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:40.770356 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.768891 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab48f110-7cc1-45fb-a416-527d9c919917-tls-certs\") pod \"test-model-63be7a-kserve-7bbf984f67-sp5cz\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:40.770356 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.769011 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-model-cache\") pod \"test-model-63be7a-kserve-7bbf984f67-sp5cz\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:40.770356 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.769070 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-dshm\") pod \"test-model-63be7a-kserve-7bbf984f67-sp5cz\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:40.770356 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.769239 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-tmp-dir\") pod \"test-model-63be7a-kserve-7bbf984f67-sp5cz\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:40.771648 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.771082 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-tmp-dir\") pod \"test-model-63be7a-kserve-7bbf984f67-sp5cz\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:40.771648 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.771361 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-model-cache\") pod \"test-model-63be7a-kserve-7bbf984f67-sp5cz\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:40.774169 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.772014 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-home\") pod \"test-model-63be7a-kserve-7bbf984f67-sp5cz\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:40.788677 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.788646 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-dshm\") pod \"test-model-63be7a-kserve-7bbf984f67-sp5cz\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:40.789032 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.789010 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab48f110-7cc1-45fb-a416-527d9c919917-tls-certs\") pod \"test-model-63be7a-kserve-7bbf984f67-sp5cz\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:40.789332 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.789302 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2db6\" (UniqueName: \"kubernetes.io/projected/ab48f110-7cc1-45fb-a416-527d9c919917-kube-api-access-n2db6\") pod \"test-model-63be7a-kserve-7bbf984f67-sp5cz\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:40.957953 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:40.957915 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:41.341498 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:17:41.341465 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab48f110_7cc1_45fb_a416_527d9c919917.slice/crio-2578802dfad3764d3b397ceb120047d315abca111b35c3c27ffe1e465f13832d WatchSource:0}: Error finding container 2578802dfad3764d3b397ceb120047d315abca111b35c3c27ffe1e465f13832d: Status 404 returned error can't find the container with id 2578802dfad3764d3b397ceb120047d315abca111b35c3c27ffe1e465f13832d Nov 02 18:17:41.341734 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:41.341715 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz"] Nov 02 18:17:42.200079 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:42.200032 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" event={"ID":"ab48f110-7cc1-45fb-a416-527d9c919917","Type":"ContainerStarted","Data":"6072c6b790ec75e21fa28a40f911d33b75fcc92421d3dfa2b167d3985c3c9e21"} Nov 02 18:17:42.200571 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:42.200096 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" event={"ID":"ab48f110-7cc1-45fb-a416-527d9c919917","Type":"ContainerStarted","Data":"2578802dfad3764d3b397ceb120047d315abca111b35c3c27ffe1e465f13832d"} Nov 02 18:17:42.227144 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:42.227093 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" podStartSLOduration=2.227078619 podStartE2EDuration="2.227078619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:42.220755218 +0000 UTC m=+2626.911809243" watchObservedRunningTime="2026-11-02 18:17:42.227078619 +0000 UTC m=+2626.918132696" Nov 02 18:17:42.958739 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:42.958691 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:50.959282 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:50.959252 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:50.976510 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:50.976484 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:51.298029 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:51.297952 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:17:58.619807 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:17:58.619360 5601 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:18:04.486816 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:18:04.486780 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:18:04.630758 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:04.630726 5601 patch_prober.go:29] interesting pod/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:18:04.630918 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:04.630789 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" podUID="5c2aa306-9a80-4299-9ada-d7229e813dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:18:08.548920 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:08.548887 5601 generic.go:356] "Generic (PLEG): container finished" podID="5c2aa306-9a80-4299-9ada-d7229e813dee" containerID="e931d5cf9949b1434fbe01a656b3bbc428c5257df06f89676106190f88818a29" exitCode=0 Nov 02 18:18:08.549407 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:08.548958 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" event={"ID":"5c2aa306-9a80-4299-9ada-d7229e813dee","Type":"ContainerDied","Data":"e931d5cf9949b1434fbe01a656b3bbc428c5257df06f89676106190f88818a29"} Nov 02 18:18:08.549527 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:08.549412 5601 scope.go:122] "RemoveContainer" containerID="e931d5cf9949b1434fbe01a656b3bbc428c5257df06f89676106190f88818a29" Nov 02 18:18:08.551361 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:08.551330 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:18:08.551458 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:08.551406 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:18:08.551458 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:08.551440 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:18:09.559108 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:09.559068 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" event={"ID":"5c2aa306-9a80-4299-9ada-d7229e813dee","Type":"ContainerStarted","Data":"c09fba55cae07e47a4831733ab1d8cb7d959a4b8a25dd20d60574f366aff63a4"} Nov 02 18:18:09.808827 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:09.808793 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:18:09.809019 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:09.809002 5601 patch_prober.go:29] interesting pod/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 02 18:18:09.809066 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:09.809040 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" podUID="5c2aa306-9a80-4299-9ada-d7229e813dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 02 18:18:10.809047 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:10.809012 5601 patch_prober.go:29] interesting pod/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 02 18:18:10.809475 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:10.809065 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" podUID="5c2aa306-9a80-4299-9ada-d7229e813dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 02 18:18:11.823726 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:11.812952 5601 patch_prober.go:29] interesting pod/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 02 18:18:11.823726 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:11.813035 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" podUID="5c2aa306-9a80-4299-9ada-d7229e813dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 02 18:18:12.809233 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:12.809198 5601 patch_prober.go:29] interesting pod/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 02 18:18:12.809427 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:12.809248 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" podUID="5c2aa306-9a80-4299-9ada-d7229e813dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 02 18:18:13.809827 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:13.809792 5601 patch_prober.go:29] interesting pod/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 02 18:18:13.810342 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:13.810298 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" podUID="5c2aa306-9a80-4299-9ada-d7229e813dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 02 18:18:14.810641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:14.810436 5601 patch_prober.go:29] interesting pod/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 02 18:18:14.810641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:14.810512 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" podUID="5c2aa306-9a80-4299-9ada-d7229e813dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 02 18:18:15.809118 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:15.809080 5601 patch_prober.go:29] interesting pod/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 02 18:18:15.809299 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:15.809131 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" podUID="5c2aa306-9a80-4299-9ada-d7229e813dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 02 18:18:16.814726 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:16.814690 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:18:16.815206 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:16.815183 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:18:16.816003 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:16.815973 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:18:34.488841 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:18:34.488772 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:18:36.874708 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:36.874669 5601 generic.go:356] "Generic (PLEG): container finished" podID="ea7a3828-ae1f-4f34-a847-3a03b824cd33" containerID="a047143d12c6314cf0662fbff8e7c329aced046d5435f2e7e0c0bb62585ea600" exitCode=0 Nov 02 18:18:36.875274 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:36.874744 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" event={"ID":"ea7a3828-ae1f-4f34-a847-3a03b824cd33","Type":"ContainerDied","Data":"a047143d12c6314cf0662fbff8e7c329aced046d5435f2e7e0c0bb62585ea600"} Nov 02 18:18:36.875350 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:36.875323 5601 scope.go:122] "RemoveContainer" containerID="a047143d12c6314cf0662fbff8e7c329aced046d5435f2e7e0c0bb62585ea600" Nov 02 18:18:36.877386 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:36.877356 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:18:36.877473 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:36.877428 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:18:36.877473 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:36.877463 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:18:37.884076 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:37.884042 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" event={"ID":"ea7a3828-ae1f-4f34-a847-3a03b824cd33","Type":"ContainerStarted","Data":"c21fe46f43104e7cea2eddc5868ed020b61ddcb1de8aa145813324570d783b23"} Nov 02 18:18:37.958768 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:37.950940 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:18:38.550922 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:38.550363 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:18:38.564688 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:38.552785 5601 patch_prober.go:29] interesting pod/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 02 18:18:38.564688 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:38.552863 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" podUID="ea7a3828-ae1f-4f34-a847-3a03b824cd33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 02 18:18:39.556819 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:39.556780 5601 patch_prober.go:29] interesting pod/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 02 18:18:39.557428 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:39.557396 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" podUID="ea7a3828-ae1f-4f34-a847-3a03b824cd33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 02 18:18:40.554382 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:40.554342 5601 patch_prober.go:29] interesting pod/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 02 18:18:40.554730 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:40.554694 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" podUID="ea7a3828-ae1f-4f34-a847-3a03b824cd33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 02 18:18:41.559083 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:41.559041 5601 patch_prober.go:29] interesting pod/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 02 18:18:41.559534 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:41.559113 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" podUID="ea7a3828-ae1f-4f34-a847-3a03b824cd33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 02 18:18:42.549102 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:42.549060 5601 patch_prober.go:29] interesting pod/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 02 18:18:42.549429 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:42.549393 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" podUID="ea7a3828-ae1f-4f34-a847-3a03b824cd33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 02 18:18:43.545424 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:43.545392 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:18:43.546448 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:18:43.546424 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:19:04.489163 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:19:04.489117 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:19:34.486460 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:19:34.486422 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:20:04.488668 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:20:04.488631 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:20:04.553412 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.553373 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7"] Nov 02 18:20:04.560345 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.560319 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:04.570784 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.570750 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-50b3a6\"/\"kube-root-ca.crt\"" Nov 02 18:20:04.571947 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.571911 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-50b3a6\"/\"openshift-service-ca.crt\"" Nov 02 18:20:04.572317 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.572299 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-50b3a6\"/\"test-model-50b3a6-kserve-self-signed-certs\"" Nov 02 18:20:04.572551 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.572536 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-50b3a6\"/\"default-dockercfg-5r6mt\"" Nov 02 18:20:04.573005 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.572984 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7"] Nov 02 18:20:04.663651 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.663575 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-tmp-dir\") pod \"test-model-50b3a6-kserve-68644d68d5-rlhr7\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:04.663836 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.663677 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-home\") pod \"test-model-50b3a6-kserve-68644d68d5-rlhr7\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:04.663836 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.663765 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-model-cache\") pod \"test-model-50b3a6-kserve-68644d68d5-rlhr7\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:04.663977 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.663866 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl62d\" (UniqueName: \"kubernetes.io/projected/dd384d95-642a-4f20-9981-4dc6eb903c7f-kube-api-access-zl62d\") pod \"test-model-50b3a6-kserve-68644d68d5-rlhr7\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:04.664088 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.664069 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd384d95-642a-4f20-9981-4dc6eb903c7f-tls-certs\") pod \"test-model-50b3a6-kserve-68644d68d5-rlhr7\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:04.664199 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.664135 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-dshm\") pod \"test-model-50b3a6-kserve-68644d68d5-rlhr7\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:04.779307 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.777969 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd384d95-642a-4f20-9981-4dc6eb903c7f-tls-certs\") pod \"test-model-50b3a6-kserve-68644d68d5-rlhr7\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:04.779307 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.778056 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-dshm\") pod \"test-model-50b3a6-kserve-68644d68d5-rlhr7\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:04.779307 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.778170 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-tmp-dir\") pod \"test-model-50b3a6-kserve-68644d68d5-rlhr7\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:04.779307 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.778192 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-home\") pod \"test-model-50b3a6-kserve-68644d68d5-rlhr7\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:04.779307 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.778269 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-model-cache\") pod \"test-model-50b3a6-kserve-68644d68d5-rlhr7\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:04.779307 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.778359 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl62d\" (UniqueName: \"kubernetes.io/projected/dd384d95-642a-4f20-9981-4dc6eb903c7f-kube-api-access-zl62d\") pod \"test-model-50b3a6-kserve-68644d68d5-rlhr7\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:04.781235 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.780935 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-tmp-dir\") pod \"test-model-50b3a6-kserve-68644d68d5-rlhr7\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:04.797697 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.781685 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-home\") pod \"test-model-50b3a6-kserve-68644d68d5-rlhr7\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:04.797697 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.781934 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-model-cache\") pod \"test-model-50b3a6-kserve-68644d68d5-rlhr7\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:04.820994 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.820908 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-dshm\") pod \"test-model-50b3a6-kserve-68644d68d5-rlhr7\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:04.841436 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.841360 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl62d\" (UniqueName: \"kubernetes.io/projected/dd384d95-642a-4f20-9981-4dc6eb903c7f-kube-api-access-zl62d\") pod \"test-model-50b3a6-kserve-68644d68d5-rlhr7\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:04.847710 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.847606 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd384d95-642a-4f20-9981-4dc6eb903c7f-tls-certs\") pod \"test-model-50b3a6-kserve-68644d68d5-rlhr7\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:04.885642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:04.884399 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:05.294264 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:05.294229 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7"] Nov 02 18:20:05.943231 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:05.943198 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" event={"ID":"dd384d95-642a-4f20-9981-4dc6eb903c7f","Type":"ContainerStarted","Data":"5f2492066dbf75ce2e382c3cf5f9fb7549b489831a6bd8b67bf49e17b93adb12"} Nov 02 18:20:05.943687 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:05.943239 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" event={"ID":"dd384d95-642a-4f20-9981-4dc6eb903c7f","Type":"ContainerStarted","Data":"5040d10c71b5d792fa2885f252b05455384d8fbac4dd0e5a7336cbed9b96e69a"} Nov 02 18:20:05.975771 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:05.975713 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" podStartSLOduration=1.9756979129999999 podStartE2EDuration="1.975697913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:20:05.972694443 +0000 UTC m=+2770.663748455" watchObservedRunningTime="2026-11-02 18:20:05.975697913 +0000 UTC m=+2770.666751925" Nov 02 18:20:06.885044 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:06.885011 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:14.885093 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:14.885053 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:14.898726 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:14.898700 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:15.091697 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:20:15.091663 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:20:34.488005 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:20:34.487965 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:21:04.487694 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:21:04.487596 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:21:32.199662 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:32.199605 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl"] Nov 02 18:21:32.204482 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:32.204437 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" podUID="84ed3fb1-d6b4-469c-b7a3-8af367d47208" containerName="istio-proxy" containerID="cri-o://fb76f1efc01f81666475fa42d6dac18bffc3ce59e041c6b4bc71f4f844a208fa" gracePeriod=30 Nov 02 18:21:34.498336 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:21:34.496878 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:21:37.010338 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.010305 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m"] Nov 02 18:21:37.015165 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.015139 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" Nov 02 18:21:37.025603 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.025572 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m"] Nov 02 18:21:37.026341 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.025899 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2966e7\"" Nov 02 18:21:37.026341 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.026029 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2966e7-dockercfg-x5spn\"" Nov 02 18:21:37.047643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.047443 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln6g8\" (UniqueName: \"kubernetes.io/projected/993ea19b-d958-4efa-91c6-e9ba381b23ef-kube-api-access-ln6g8\") pod \"payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m\" (UID: \"993ea19b-d958-4efa-91c6-e9ba381b23ef\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" Nov 02 18:21:37.047643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.047564 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993ea19b-d958-4efa-91c6-e9ba381b23ef-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m\" (UID: \"993ea19b-d958-4efa-91c6-e9ba381b23ef\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" Nov 02 18:21:37.238640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.233035 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993ea19b-d958-4efa-91c6-e9ba381b23ef-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m\" (UID: \"993ea19b-d958-4efa-91c6-e9ba381b23ef\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" Nov 02 18:21:37.238640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.233425 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln6g8\" (UniqueName: \"kubernetes.io/projected/993ea19b-d958-4efa-91c6-e9ba381b23ef-kube-api-access-ln6g8\") pod \"payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m\" (UID: \"993ea19b-d958-4efa-91c6-e9ba381b23ef\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" Nov 02 18:21:37.238640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.234740 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993ea19b-d958-4efa-91c6-e9ba381b23ef-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m\" (UID: \"993ea19b-d958-4efa-91c6-e9ba381b23ef\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" Nov 02 18:21:37.295376 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.295266 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln6g8\" (UniqueName: \"kubernetes.io/projected/993ea19b-d958-4efa-91c6-e9ba381b23ef-kube-api-access-ln6g8\") pod \"payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m\" (UID: \"993ea19b-d958-4efa-91c6-e9ba381b23ef\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" Nov 02 18:21:37.339194 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.338929 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" Nov 02 18:21:37.650876 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.650843 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:21:37.691876 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.690365 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84"] Nov 02 18:21:37.697474 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:21:37.697396 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993ea19b_d958_4efa_91c6_e9ba381b23ef.slice/crio-f4a60ebf8aa17b950ca82913466084ca84ecaa8b86f73799713003fbdf78ed68 WatchSource:0}: Error finding container f4a60ebf8aa17b950ca82913466084ca84ecaa8b86f73799713003fbdf78ed68: Status 404 returned error can't find the container with id f4a60ebf8aa17b950ca82913466084ca84ecaa8b86f73799713003fbdf78ed68 Nov 02 18:21:37.699159 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.699096 5601 cpu_manager.go:414] "containerMap: removing container" podUID="84ed3fb1-d6b4-469c-b7a3-8af367d47208" containerName="istio-proxy" Nov 02 18:21:37.699159 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.699118 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84ed3fb1-d6b4-469c-b7a3-8af367d47208" containerName="istio-proxy" Nov 02 18:21:37.699426 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.699166 5601 cpu_manager.go:414] "containerMap: removing container" podUID="84ed3fb1-d6b4-469c-b7a3-8af367d47208" containerName="istio-proxy" Nov 02 18:21:37.699426 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.699176 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84ed3fb1-d6b4-469c-b7a3-8af367d47208" containerName="istio-proxy" Nov 02 18:21:37.699426 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.699346 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="84ed3fb1-d6b4-469c-b7a3-8af367d47208" containerName="istio-proxy" Nov 02 18:21:37.707108 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.707084 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m"] Nov 02 18:21:37.711978 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.710092 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.712504 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.712478 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84"] Nov 02 18:21:37.737538 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.737113 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-63be7a-openshift-default-dockercfg-5274g\"" Nov 02 18:21:37.751028 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.750984 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-envoy\") pod \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " Nov 02 18:21:37.753014 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.751252 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-credential-socket\") pod \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " Nov 02 18:21:37.753014 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.751321 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-workload-certs\") pod \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " Nov 02 18:21:37.753014 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.751412 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-token\" (UniqueName: \"kubernetes.io/projected/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-token\") pod \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " Nov 02 18:21:37.753014 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.752899 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-workload-certs" pod "84ed3fb1-d6b4-469c-b7a3-8af367d47208" (UID: "84ed3fb1-d6b4-469c-b7a3-8af367d47208"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:37.753557 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.753352 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-credential-socket" pod "84ed3fb1-d6b4-469c-b7a3-8af367d47208" (UID: "84ed3fb1-d6b4-469c-b7a3-8af367d47208"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:37.753637 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.753576 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-workload-socket\") pod \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " Nov 02 18:21:37.753701 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.753658 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istiod-ca-cert\") pod \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " Nov 02 18:21:37.753759 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.753700 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84ed3fb1-d6b4-469c-b7a3-8af367d47208-kube-api-access-sv8mn\" (UniqueName: \"kubernetes.io/projected/84ed3fb1-d6b4-469c-b7a3-8af367d47208-kube-api-access-sv8mn\") pod \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " Nov 02 18:21:37.753759 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.753729 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-data\") pod \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " Nov 02 18:21:37.753868 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.753844 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-podinfo\") pod \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\" (UID: \"84ed3fb1-d6b4-469c-b7a3-8af367d47208\") " Nov 02 18:21:37.754212 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.754187 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.754342 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.754252 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smtf9\" (UniqueName: \"kubernetes.io/projected/3b429f46-034e-43a5-bffb-65f58b9c84a4-kube-api-access-smtf9\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.754430 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.754394 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b429f46-034e-43a5-bffb-65f58b9c84a4-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.754573 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.754458 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.754573 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.754526 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.754722 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.754576 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.756761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.756539 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-envoy" pod "84ed3fb1-d6b4-469c-b7a3-8af367d47208" (UID: "84ed3fb1-d6b4-469c-b7a3-8af367d47208"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:37.757378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.757343 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istiod-ca-cert" pod "84ed3fb1-d6b4-469c-b7a3-8af367d47208" (UID: "84ed3fb1-d6b4-469c-b7a3-8af367d47208"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:37.757667 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.754607 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.757833 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.757810 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.757966 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.757947 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.758272 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.758248 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:37.758828 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.758752 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:37.758980 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.758783 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:37.758980 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.758960 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:37.758980 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.758961 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-data" pod "84ed3fb1-d6b4-469c-b7a3-8af367d47208" (UID: "84ed3fb1-d6b4-469c-b7a3-8af367d47208"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:37.759161 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.759030 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-workload-socket" pod "84ed3fb1-d6b4-469c-b7a3-8af367d47208" (UID: "84ed3fb1-d6b4-469c-b7a3-8af367d47208"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:37.760920 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.759917 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-token" pod "84ed3fb1-d6b4-469c-b7a3-8af367d47208" (UID: "84ed3fb1-d6b4-469c-b7a3-8af367d47208"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:37.761046 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.761011 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ed3fb1-d6b4-469c-b7a3-8af367d47208-kube-api-access-sv8mn" pod "84ed3fb1-d6b4-469c-b7a3-8af367d47208" (UID: "84ed3fb1-d6b4-469c-b7a3-8af367d47208"). InnerVolumeSpecName "kube-api-access-sv8mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:37.763224 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.763181 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-podinfo" pod "84ed3fb1-d6b4-469c-b7a3-8af367d47208" (UID: "84ed3fb1-d6b4-469c-b7a3-8af367d47208"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:21:37.865957 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.865919 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b429f46-034e-43a5-bffb-65f58b9c84a4-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.866405 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.866379 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.866583 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.866566 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.866740 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.866724 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.866920 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.866902 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.867187 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.867131 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.867686 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.867665 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.867898 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.867881 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.868443 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.868019 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.868443 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.867318 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b429f46-034e-43a5-bffb-65f58b9c84a4-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.868632 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.868555 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smtf9\" (UniqueName: \"kubernetes.io/projected/3b429f46-034e-43a5-bffb-65f58b9c84a4-kube-api-access-smtf9\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.870963 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.868891 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.870963 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.869171 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:37.870963 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.869216 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:37.870963 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.869234 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv8mn\" (UniqueName: \"kubernetes.io/projected/84ed3fb1-d6b4-469c-b7a3-8af367d47208-kube-api-access-sv8mn\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:37.870963 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.869251 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:37.870963 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.869265 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ed3fb1-d6b4-469c-b7a3-8af367d47208-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:37.871813 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.871442 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.871813 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.871704 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.881129 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.881046 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.882439 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.882396 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.884542 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.884519 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:37.884815 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:37.884793 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smtf9\" (UniqueName: \"kubernetes.io/projected/3b429f46-034e-43a5-bffb-65f58b9c84a4-kube-api-access-smtf9\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:38.061657 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.061612 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:38.287517 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.287481 5601 generic.go:356] "Generic (PLEG): container finished" podID="84ed3fb1-d6b4-469c-b7a3-8af367d47208" containerID="fb76f1efc01f81666475fa42d6dac18bffc3ce59e041c6b4bc71f4f844a208fa" exitCode=0 Nov 02 18:21:38.288203 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.287583 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" event={"ID":"84ed3fb1-d6b4-469c-b7a3-8af367d47208","Type":"ContainerDied","Data":"fb76f1efc01f81666475fa42d6dac18bffc3ce59e041c6b4bc71f4f844a208fa"} Nov 02 18:21:38.288203 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.287665 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" event={"ID":"84ed3fb1-d6b4-469c-b7a3-8af367d47208","Type":"ContainerDied","Data":"2f41205b2beee590c57d7e6aaa8b972f5113e6822dcef6625ecb7e9a0b3e3681"} Nov 02 18:21:38.288203 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.287704 5601 scope.go:122] "RemoveContainer" containerID="fb76f1efc01f81666475fa42d6dac18bffc3ce59e041c6b4bc71f4f844a208fa" Nov 02 18:21:38.288203 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.287931 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl" Nov 02 18:21:38.333867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.332831 5601 scope.go:122] "RemoveContainer" containerID="75ce8e3cf73639828799ca59a1c1271b21f787c6c1d6803312638ee5ef5db6ff" Nov 02 18:21:38.333867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.333068 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" event={"ID":"993ea19b-d958-4efa-91c6-e9ba381b23ef","Type":"ContainerStarted","Data":"4dea66ae88b5b4f49c2bc8f3068c23e70f6e4c0491a97ce8e6eb6d5bb8b537ed"} Nov 02 18:21:38.333867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.333124 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" event={"ID":"993ea19b-d958-4efa-91c6-e9ba381b23ef","Type":"ContainerStarted","Data":"f4a60ebf8aa17b950ca82913466084ca84ecaa8b86f73799713003fbdf78ed68"} Nov 02 18:21:38.334841 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.334134 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" Nov 02 18:21:38.355519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.355448 5601 scope.go:122] "RemoveContainer" containerID="fb76f1efc01f81666475fa42d6dac18bffc3ce59e041c6b4bc71f4f844a208fa" Nov 02 18:21:38.356309 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:21:38.356280 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb76f1efc01f81666475fa42d6dac18bffc3ce59e041c6b4bc71f4f844a208fa\": container with ID starting with fb76f1efc01f81666475fa42d6dac18bffc3ce59e041c6b4bc71f4f844a208fa not found: ID does not exist" containerID="fb76f1efc01f81666475fa42d6dac18bffc3ce59e041c6b4bc71f4f844a208fa" Nov 02 18:21:38.356434 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.356324 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb76f1efc01f81666475fa42d6dac18bffc3ce59e041c6b4bc71f4f844a208fa"} err="failed to get container status \"fb76f1efc01f81666475fa42d6dac18bffc3ce59e041c6b4bc71f4f844a208fa\": rpc error: code = NotFound desc = could not find container \"fb76f1efc01f81666475fa42d6dac18bffc3ce59e041c6b4bc71f4f844a208fa\": container with ID starting with fb76f1efc01f81666475fa42d6dac18bffc3ce59e041c6b4bc71f4f844a208fa not found: ID does not exist" Nov 02 18:21:38.356434 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.356351 5601 scope.go:122] "RemoveContainer" containerID="75ce8e3cf73639828799ca59a1c1271b21f787c6c1d6803312638ee5ef5db6ff" Nov 02 18:21:38.356762 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:21:38.356733 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ce8e3cf73639828799ca59a1c1271b21f787c6c1d6803312638ee5ef5db6ff\": container with ID starting with 75ce8e3cf73639828799ca59a1c1271b21f787c6c1d6803312638ee5ef5db6ff not found: ID does not exist" containerID="75ce8e3cf73639828799ca59a1c1271b21f787c6c1d6803312638ee5ef5db6ff" Nov 02 18:21:38.356865 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.356772 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ce8e3cf73639828799ca59a1c1271b21f787c6c1d6803312638ee5ef5db6ff"} err="failed to get container status \"75ce8e3cf73639828799ca59a1c1271b21f787c6c1d6803312638ee5ef5db6ff\": rpc error: code = NotFound desc = could not find container \"75ce8e3cf73639828799ca59a1c1271b21f787c6c1d6803312638ee5ef5db6ff\": container with ID starting with 75ce8e3cf73639828799ca59a1c1271b21f787c6c1d6803312638ee5ef5db6ff not found: ID does not exist" Nov 02 18:21:38.359004 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.358959 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" podStartSLOduration=2.3589476129999998 podStartE2EDuration="2.358947613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:38.355733878 +0000 UTC m=+2863.046787890" watchObservedRunningTime="2026-11-02 18:21:38.358947613 +0000 UTC m=+2863.050001626" Nov 02 18:21:38.373057 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.372977 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84"] Nov 02 18:21:38.374441 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:21:38.374199 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b429f46_034e_43a5_bffb_65f58b9c84a4.slice/crio-342b671ae45e65bd10856367a5c4f0bfe524dbc307fc433e7af1fba802baed13 WatchSource:0}: Error finding container 342b671ae45e65bd10856367a5c4f0bfe524dbc307fc433e7af1fba802baed13: Status 404 returned error can't find the container with id 342b671ae45e65bd10856367a5c4f0bfe524dbc307fc433e7af1fba802baed13 Nov 02 18:21:38.378897 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.378875 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl"] Nov 02 18:21:38.384013 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.383989 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-658b8c-openshift-default-777bdf6b85-srgrl"] Nov 02 18:21:38.384817 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.384785 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:38.384932 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.384845 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:38.384932 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:38.384875 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:21:39.359984 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:39.359949 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" event={"ID":"3b429f46-034e-43a5-bffb-65f58b9c84a4","Type":"ContainerStarted","Data":"2c6ab9f2831902ede4cb641a12205663b57da652786eb15d6897b6b47884b9ff"} Nov 02 18:21:39.360430 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:39.359999 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" event={"ID":"3b429f46-034e-43a5-bffb-65f58b9c84a4","Type":"ContainerStarted","Data":"342b671ae45e65bd10856367a5c4f0bfe524dbc307fc433e7af1fba802baed13"} Nov 02 18:21:39.370716 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:39.370682 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd"] Nov 02 18:21:39.370879 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:39.370860 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" podUID="e162853a-15f2-42ad-b350-bbc126c32272" containerName="istio-proxy" containerID="cri-o://e629be86004f42bd7130d861f15eb0e0dbd3508c75fbede6e774784cd052e409" gracePeriod=30 Nov 02 18:21:39.403053 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:39.402989 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" podStartSLOduration=101.402969836 podStartE2EDuration="1m41.402969836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:39.385000617 +0000 UTC m=+2864.076054641" watchObservedRunningTime="2026-11-02 18:21:39.402969836 +0000 UTC m=+2864.094023849" Nov 02 18:21:39.463745 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:39.463701 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84ed3fb1-d6b4-469c-b7a3-8af367d47208" path="/var/lib/kubelet/pods/84ed3fb1-d6b4-469c-b7a3-8af367d47208/volumes" Nov 02 18:21:40.062633 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:40.062583 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:40.064541 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:40.064519 5601 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 02 18:21:40.064608 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:40.064567 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" podUID="3b429f46-034e-43a5-bffb-65f58b9c84a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 02 18:21:40.414407 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:40.414329 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499"] Nov 02 18:21:40.414871 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:40.414558 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" podUID="a15cd03a-9403-4912-a060-4ae6dab8baf8" containerName="istio-proxy" containerID="cri-o://d1afba8649d8a88565cd250391a5f4e01b6870473ca0819b56d2b8cf6c1e5d9e" gracePeriod=30 Nov 02 18:21:41.062356 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:41.062317 5601 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 02 18:21:41.062551 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:41.062384 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" podUID="3b429f46-034e-43a5-bffb-65f58b9c84a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 02 18:21:42.062135 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:42.062085 5601 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 02 18:21:42.062571 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:42.062154 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" podUID="3b429f46-034e-43a5-bffb-65f58b9c84a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 02 18:21:43.083670 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:43.080193 5601 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 02 18:21:43.083670 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:43.080257 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" podUID="3b429f46-034e-43a5-bffb-65f58b9c84a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 02 18:21:44.062866 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.062818 5601 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 02 18:21:44.063062 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.062910 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" podUID="3b429f46-034e-43a5-bffb-65f58b9c84a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 02 18:21:44.463788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.463744 5601 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 18:21:44.464261 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.463819 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" podUID="e162853a-15f2-42ad-b350-bbc126c32272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 18:21:44.789728 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.789327 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr"] Nov 02 18:21:44.798246 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.798011 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 18:21:44.922599 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.922237 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e162853a-15f2-42ad-b350-bbc126c32272-istio-token\" (UniqueName: \"kubernetes.io/projected/e162853a-15f2-42ad-b350-bbc126c32272-istio-token\") pod \"e162853a-15f2-42ad-b350-bbc126c32272\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " Nov 02 18:21:44.922599 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.922307 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e162853a-15f2-42ad-b350-bbc126c32272-kube-api-access-p45p8\" (UniqueName: \"kubernetes.io/projected/e162853a-15f2-42ad-b350-bbc126c32272-kube-api-access-p45p8\") pod \"e162853a-15f2-42ad-b350-bbc126c32272\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " Nov 02 18:21:44.922599 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.922371 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-istio-data\") pod \"e162853a-15f2-42ad-b350-bbc126c32272\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " Nov 02 18:21:44.922599 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.922484 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-credential-socket\") pod \"e162853a-15f2-42ad-b350-bbc126c32272\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " Nov 02 18:21:44.922599 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.922531 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-workload-certs\") pod \"e162853a-15f2-42ad-b350-bbc126c32272\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " Nov 02 18:21:44.922599 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.922639 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e162853a-15f2-42ad-b350-bbc126c32272-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e162853a-15f2-42ad-b350-bbc126c32272-istio-podinfo\") pod \"e162853a-15f2-42ad-b350-bbc126c32272\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " Nov 02 18:21:44.923249 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.922710 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-istio-envoy\") pod \"e162853a-15f2-42ad-b350-bbc126c32272\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " Nov 02 18:21:44.923249 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.922803 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e162853a-15f2-42ad-b350-bbc126c32272-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e162853a-15f2-42ad-b350-bbc126c32272-istiod-ca-cert\") pod \"e162853a-15f2-42ad-b350-bbc126c32272\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " Nov 02 18:21:44.923249 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.922859 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-workload-socket\") pod \"e162853a-15f2-42ad-b350-bbc126c32272\" (UID: \"e162853a-15f2-42ad-b350-bbc126c32272\") " Nov 02 18:21:44.923533 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.923504 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-credential-socket" pod "e162853a-15f2-42ad-b350-bbc126c32272" (UID: "e162853a-15f2-42ad-b350-bbc126c32272"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:44.923910 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.923882 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-workload-certs" pod "e162853a-15f2-42ad-b350-bbc126c32272" (UID: "e162853a-15f2-42ad-b350-bbc126c32272"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:44.924359 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.924319 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-istio-data" pod "e162853a-15f2-42ad-b350-bbc126c32272" (UID: "e162853a-15f2-42ad-b350-bbc126c32272"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:44.924737 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.924686 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e162853a-15f2-42ad-b350-bbc126c32272-istiod-ca-cert" pod "e162853a-15f2-42ad-b350-bbc126c32272" (UID: "e162853a-15f2-42ad-b350-bbc126c32272"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:44.925007 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.924977 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-workload-socket" pod "e162853a-15f2-42ad-b350-bbc126c32272" (UID: "e162853a-15f2-42ad-b350-bbc126c32272"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:44.926169 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.926142 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e162853a-15f2-42ad-b350-bbc126c32272-istio-token" pod "e162853a-15f2-42ad-b350-bbc126c32272" (UID: "e162853a-15f2-42ad-b350-bbc126c32272"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:44.927661 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.927599 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-istio-envoy" pod "e162853a-15f2-42ad-b350-bbc126c32272" (UID: "e162853a-15f2-42ad-b350-bbc126c32272"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:44.930665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.928219 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e162853a-15f2-42ad-b350-bbc126c32272-istio-podinfo" pod "e162853a-15f2-42ad-b350-bbc126c32272" (UID: "e162853a-15f2-42ad-b350-bbc126c32272"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:21:44.930665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:44.929976 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e162853a-15f2-42ad-b350-bbc126c32272-kube-api-access-p45p8" pod "e162853a-15f2-42ad-b350-bbc126c32272" (UID: "e162853a-15f2-42ad-b350-bbc126c32272"). InnerVolumeSpecName "kube-api-access-p45p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:45.025825 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.025740 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e162853a-15f2-42ad-b350-bbc126c32272-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:45.025825 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.025779 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:45.025825 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.025793 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e162853a-15f2-42ad-b350-bbc126c32272-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:45.025825 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.025805 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:45.025825 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.025820 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e162853a-15f2-42ad-b350-bbc126c32272-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:45.025825 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.025834 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p45p8\" (UniqueName: \"kubernetes.io/projected/e162853a-15f2-42ad-b350-bbc126c32272-kube-api-access-p45p8\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:45.026237 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.025848 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:45.026237 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.025863 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:45.026237 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.025875 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e162853a-15f2-42ad-b350-bbc126c32272-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:45.062737 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.062702 5601 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 02 18:21:45.062918 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.062764 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" podUID="3b429f46-034e-43a5-bffb-65f58b9c84a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 02 18:21:45.475640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.470248 5601 generic.go:356] "Generic (PLEG): container finished" podID="e162853a-15f2-42ad-b350-bbc126c32272" containerID="e629be86004f42bd7130d861f15eb0e0dbd3508c75fbede6e774784cd052e409" exitCode=0 Nov 02 18:21:45.475640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.470368 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" event={"ID":"e162853a-15f2-42ad-b350-bbc126c32272","Type":"ContainerDied","Data":"e629be86004f42bd7130d861f15eb0e0dbd3508c75fbede6e774784cd052e409"} Nov 02 18:21:45.475640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.470415 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" event={"ID":"e162853a-15f2-42ad-b350-bbc126c32272","Type":"ContainerDied","Data":"eb7f0a62777c022a45b7c70260defb6f840a9b05fb4c584e4bcf829f21d26472"} Nov 02 18:21:45.475640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.470445 5601 scope.go:122] "RemoveContainer" containerID="e629be86004f42bd7130d861f15eb0e0dbd3508c75fbede6e774784cd052e409" Nov 02 18:21:45.475640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.470599 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd" Nov 02 18:21:45.504296 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.503895 5601 scope.go:122] "RemoveContainer" containerID="e629be86004f42bd7130d861f15eb0e0dbd3508c75fbede6e774784cd052e409" Nov 02 18:21:45.505373 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:21:45.505241 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e629be86004f42bd7130d861f15eb0e0dbd3508c75fbede6e774784cd052e409\": container with ID starting with e629be86004f42bd7130d861f15eb0e0dbd3508c75fbede6e774784cd052e409 not found: ID does not exist" containerID="e629be86004f42bd7130d861f15eb0e0dbd3508c75fbede6e774784cd052e409" Nov 02 18:21:45.505373 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.505288 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e629be86004f42bd7130d861f15eb0e0dbd3508c75fbede6e774784cd052e409"} err="failed to get container status \"e629be86004f42bd7130d861f15eb0e0dbd3508c75fbede6e774784cd052e409\": rpc error: code = NotFound desc = could not find container \"e629be86004f42bd7130d861f15eb0e0dbd3508c75fbede6e774784cd052e409\": container with ID starting with e629be86004f42bd7130d861f15eb0e0dbd3508c75fbede6e774784cd052e409 not found: ID does not exist" Nov 02 18:21:45.589740 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.589701 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd"] Nov 02 18:21:45.591819 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.591791 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dwtbd"] Nov 02 18:21:45.772255 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.772215 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 18:21:45.858447 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.858404 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-data\") pod \"a15cd03a-9403-4912-a060-4ae6dab8baf8\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " Nov 02 18:21:45.858651 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.858502 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15cd03a-9403-4912-a060-4ae6dab8baf8-kube-api-access-d4qk8\" (UniqueName: \"kubernetes.io/projected/a15cd03a-9403-4912-a060-4ae6dab8baf8-kube-api-access-d4qk8\") pod \"a15cd03a-9403-4912-a060-4ae6dab8baf8\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " Nov 02 18:21:45.858651 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.858534 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-workload-socket\") pod \"a15cd03a-9403-4912-a060-4ae6dab8baf8\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " Nov 02 18:21:45.858782 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.858688 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a15cd03a-9403-4912-a060-4ae6dab8baf8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15cd03a-9403-4912-a060-4ae6dab8baf8-istiod-ca-cert\") pod \"a15cd03a-9403-4912-a060-4ae6dab8baf8\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " Nov 02 18:21:45.858782 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.858700 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-data" pod "a15cd03a-9403-4912-a060-4ae6dab8baf8" (UID: "a15cd03a-9403-4912-a060-4ae6dab8baf8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:45.858782 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.858772 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-credential-socket\") pod \"a15cd03a-9403-4912-a060-4ae6dab8baf8\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " Nov 02 18:21:45.858945 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.858834 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-envoy\") pod \"a15cd03a-9403-4912-a060-4ae6dab8baf8\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " Nov 02 18:21:45.858998 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.858948 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-workload-certs\") pod \"a15cd03a-9403-4912-a060-4ae6dab8baf8\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " Nov 02 18:21:45.859059 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.859043 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-token\" (UniqueName: \"kubernetes.io/projected/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-token\") pod \"a15cd03a-9403-4912-a060-4ae6dab8baf8\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " Nov 02 18:21:45.859126 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.859109 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-podinfo\") pod \"a15cd03a-9403-4912-a060-4ae6dab8baf8\" (UID: \"a15cd03a-9403-4912-a060-4ae6dab8baf8\") " Nov 02 18:21:45.859228 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.859205 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15cd03a-9403-4912-a060-4ae6dab8baf8-istiod-ca-cert" pod "a15cd03a-9403-4912-a060-4ae6dab8baf8" (UID: "a15cd03a-9403-4912-a060-4ae6dab8baf8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:45.859505 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.859484 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-workload-certs" pod "a15cd03a-9403-4912-a060-4ae6dab8baf8" (UID: "a15cd03a-9403-4912-a060-4ae6dab8baf8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:45.860026 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.859944 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:45.860026 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.859973 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:45.860026 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.859989 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15cd03a-9403-4912-a060-4ae6dab8baf8-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:45.860807 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.860777 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-credential-socket" pod "a15cd03a-9403-4912-a060-4ae6dab8baf8" (UID: "a15cd03a-9403-4912-a060-4ae6dab8baf8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:45.861102 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.861072 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-workload-socket" pod "a15cd03a-9403-4912-a060-4ae6dab8baf8" (UID: "a15cd03a-9403-4912-a060-4ae6dab8baf8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:45.862602 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.862573 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-podinfo" pod "a15cd03a-9403-4912-a060-4ae6dab8baf8" (UID: "a15cd03a-9403-4912-a060-4ae6dab8baf8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:21:45.863436 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.863399 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-token" pod "a15cd03a-9403-4912-a060-4ae6dab8baf8" (UID: "a15cd03a-9403-4912-a060-4ae6dab8baf8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:45.863757 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.863724 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15cd03a-9403-4912-a060-4ae6dab8baf8-kube-api-access-d4qk8" pod "a15cd03a-9403-4912-a060-4ae6dab8baf8" (UID: "a15cd03a-9403-4912-a060-4ae6dab8baf8"). InnerVolumeSpecName "kube-api-access-d4qk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:45.863842 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.863800 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-envoy" pod "a15cd03a-9403-4912-a060-4ae6dab8baf8" (UID: "a15cd03a-9403-4912-a060-4ae6dab8baf8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:45.962178 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.962137 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4qk8\" (UniqueName: \"kubernetes.io/projected/a15cd03a-9403-4912-a060-4ae6dab8baf8-kube-api-access-d4qk8\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:45.962178 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.962179 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:45.962426 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.962195 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:45.962426 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.962209 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:45.962426 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.962222 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:45.962426 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:45.962238 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15cd03a-9403-4912-a060-4ae6dab8baf8-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:46.490011 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:46.489969 5601 generic.go:356] "Generic (PLEG): container finished" podID="a15cd03a-9403-4912-a060-4ae6dab8baf8" containerID="d1afba8649d8a88565cd250391a5f4e01b6870473ca0819b56d2b8cf6c1e5d9e" exitCode=0 Nov 02 18:21:46.490511 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:46.490039 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" Nov 02 18:21:46.490511 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:46.490259 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" event={"ID":"a15cd03a-9403-4912-a060-4ae6dab8baf8","Type":"ContainerDied","Data":"d1afba8649d8a88565cd250391a5f4e01b6870473ca0819b56d2b8cf6c1e5d9e"} Nov 02 18:21:46.490511 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:46.490310 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499" event={"ID":"a15cd03a-9403-4912-a060-4ae6dab8baf8","Type":"ContainerDied","Data":"0d6323c67d2efb07b89c0c65b66f899ac7a4c6fa1e42fc2c6ab242fdd7f86464"} Nov 02 18:21:46.490511 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:46.490343 5601 scope.go:122] "RemoveContainer" containerID="d1afba8649d8a88565cd250391a5f4e01b6870473ca0819b56d2b8cf6c1e5d9e" Nov 02 18:21:46.518910 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:46.518743 5601 scope.go:122] "RemoveContainer" containerID="d1afba8649d8a88565cd250391a5f4e01b6870473ca0819b56d2b8cf6c1e5d9e" Nov 02 18:21:46.519079 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:21:46.519053 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1afba8649d8a88565cd250391a5f4e01b6870473ca0819b56d2b8cf6c1e5d9e\": container with ID starting with d1afba8649d8a88565cd250391a5f4e01b6870473ca0819b56d2b8cf6c1e5d9e not found: ID does not exist" containerID="d1afba8649d8a88565cd250391a5f4e01b6870473ca0819b56d2b8cf6c1e5d9e" Nov 02 18:21:46.519132 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:46.519088 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1afba8649d8a88565cd250391a5f4e01b6870473ca0819b56d2b8cf6c1e5d9e"} err="failed to get container status \"d1afba8649d8a88565cd250391a5f4e01b6870473ca0819b56d2b8cf6c1e5d9e\": rpc error: code = NotFound desc = could not find container \"d1afba8649d8a88565cd250391a5f4e01b6870473ca0819b56d2b8cf6c1e5d9e\": container with ID starting with d1afba8649d8a88565cd250391a5f4e01b6870473ca0819b56d2b8cf6c1e5d9e not found: ID does not exist" Nov 02 18:21:46.530727 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:46.530701 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499"] Nov 02 18:21:46.539346 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:46.539315 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-d7499"] Nov 02 18:21:46.562990 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:46.562954 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:46.563386 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:46.563363 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:46.564462 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:46.564431 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:21:47.435022 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:47.434987 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a15cd03a-9403-4912-a060-4ae6dab8baf8" path="/var/lib/kubelet/pods/a15cd03a-9403-4912-a060-4ae6dab8baf8/volumes" Nov 02 18:21:47.435635 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:47.435596 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e162853a-15f2-42ad-b350-bbc126c32272" path="/var/lib/kubelet/pods/e162853a-15f2-42ad-b350-bbc126c32272/volumes" Nov 02 18:21:49.790542 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:49.790500 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" podUID="1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5" containerName="payload-pre-processing" containerID="cri-o://ca796f7acc8a4f956cc40f86d3bcbf388a1b9ec2f56c6a2d5cee601c41f64bd8" gracePeriod=25 Nov 02 18:21:50.382479 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:50.382450 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" Nov 02 18:21:52.218497 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:52.218461 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz"] Nov 02 18:21:52.219199 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:52.218721 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" podUID="945a032d-c69f-4d9e-96a5-db1ac483af67" containerName="istio-proxy" containerID="cri-o://61b59d47ca148c52949dc9e30caf2d91fe8cbdfd630f3e1e387b26f89a060648" gracePeriod=30 Nov 02 18:21:55.730916 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:55.730880 5601 patch_prober.go:29] interesting pod/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:21:55.731337 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:55.730935 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" podUID="945a032d-c69f-4d9e-96a5-db1ac483af67" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:21:57.512423 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.512397 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:21:57.644398 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.644363 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-workload-certs\") pod \"945a032d-c69f-4d9e-96a5-db1ac483af67\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " Nov 02 18:21:57.644603 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.644508 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-envoy\") pod \"945a032d-c69f-4d9e-96a5-db1ac483af67\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " Nov 02 18:21:57.644603 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.644542 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-token\" (UniqueName: \"kubernetes.io/projected/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-token\") pod \"945a032d-c69f-4d9e-96a5-db1ac483af67\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " Nov 02 18:21:57.644603 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.644587 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-data\") pod \"945a032d-c69f-4d9e-96a5-db1ac483af67\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " Nov 02 18:21:57.644788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.644701 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-workload-certs" pod "945a032d-c69f-4d9e-96a5-db1ac483af67" (UID: "945a032d-c69f-4d9e-96a5-db1ac483af67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:57.644845 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.644783 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-workload-socket\") pod \"945a032d-c69f-4d9e-96a5-db1ac483af67\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " Nov 02 18:21:57.644845 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.644826 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-podinfo\") pod \"945a032d-c69f-4d9e-96a5-db1ac483af67\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " Nov 02 18:21:57.644950 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.644916 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945a032d-c69f-4d9e-96a5-db1ac483af67-kube-api-access-zjdck\" (UniqueName: \"kubernetes.io/projected/945a032d-c69f-4d9e-96a5-db1ac483af67-kube-api-access-zjdck\") pod \"945a032d-c69f-4d9e-96a5-db1ac483af67\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " Nov 02 18:21:57.645012 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.644952 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/945a032d-c69f-4d9e-96a5-db1ac483af67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945a032d-c69f-4d9e-96a5-db1ac483af67-istiod-ca-cert\") pod \"945a032d-c69f-4d9e-96a5-db1ac483af67\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " Nov 02 18:21:57.645232 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.645203 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-data" pod "945a032d-c69f-4d9e-96a5-db1ac483af67" (UID: "945a032d-c69f-4d9e-96a5-db1ac483af67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:57.645351 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.645233 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-credential-socket\") pod \"945a032d-c69f-4d9e-96a5-db1ac483af67\" (UID: \"945a032d-c69f-4d9e-96a5-db1ac483af67\") " Nov 02 18:21:57.645351 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.645274 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945a032d-c69f-4d9e-96a5-db1ac483af67-istiod-ca-cert" pod "945a032d-c69f-4d9e-96a5-db1ac483af67" (UID: "945a032d-c69f-4d9e-96a5-db1ac483af67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:57.645561 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.645538 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-credential-socket" pod "945a032d-c69f-4d9e-96a5-db1ac483af67" (UID: "945a032d-c69f-4d9e-96a5-db1ac483af67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:57.645935 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.645912 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-workload-socket" pod "945a032d-c69f-4d9e-96a5-db1ac483af67" (UID: "945a032d-c69f-4d9e-96a5-db1ac483af67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:57.646267 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.646239 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:57.646368 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.646272 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945a032d-c69f-4d9e-96a5-db1ac483af67-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:57.646368 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.646287 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:57.646368 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.646301 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:57.646368 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.646315 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:57.647132 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.647106 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-token" pod "945a032d-c69f-4d9e-96a5-db1ac483af67" (UID: "945a032d-c69f-4d9e-96a5-db1ac483af67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:57.647247 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.647177 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-envoy" pod "945a032d-c69f-4d9e-96a5-db1ac483af67" (UID: "945a032d-c69f-4d9e-96a5-db1ac483af67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:57.647572 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.647553 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-podinfo" pod "945a032d-c69f-4d9e-96a5-db1ac483af67" (UID: "945a032d-c69f-4d9e-96a5-db1ac483af67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:21:57.648249 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.648227 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945a032d-c69f-4d9e-96a5-db1ac483af67-kube-api-access-zjdck" pod "945a032d-c69f-4d9e-96a5-db1ac483af67" (UID: "945a032d-c69f-4d9e-96a5-db1ac483af67"). InnerVolumeSpecName "kube-api-access-zjdck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:57.688560 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.688525 5601 generic.go:356] "Generic (PLEG): container finished" podID="945a032d-c69f-4d9e-96a5-db1ac483af67" containerID="61b59d47ca148c52949dc9e30caf2d91fe8cbdfd630f3e1e387b26f89a060648" exitCode=0 Nov 02 18:21:57.688747 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.688579 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" Nov 02 18:21:57.688747 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.688582 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" event={"ID":"945a032d-c69f-4d9e-96a5-db1ac483af67","Type":"ContainerDied","Data":"61b59d47ca148c52949dc9e30caf2d91fe8cbdfd630f3e1e387b26f89a060648"} Nov 02 18:21:57.688747 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.688675 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz" event={"ID":"945a032d-c69f-4d9e-96a5-db1ac483af67","Type":"ContainerDied","Data":"509a31e45bc5d0fad808fbd674059c9f5272a3c559f880bc93a5d462e93eb0a9"} Nov 02 18:21:57.688747 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.688710 5601 scope.go:122] "RemoveContainer" containerID="61b59d47ca148c52949dc9e30caf2d91fe8cbdfd630f3e1e387b26f89a060648" Nov 02 18:21:57.700980 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.700962 5601 scope.go:122] "RemoveContainer" containerID="4f94841406dd48759f969b3f451b0570036eb289bfe818515e0b5e26a9451b3d" Nov 02 18:21:57.717446 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.717423 5601 scope.go:122] "RemoveContainer" containerID="61b59d47ca148c52949dc9e30caf2d91fe8cbdfd630f3e1e387b26f89a060648" Nov 02 18:21:57.717887 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:21:57.717771 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b59d47ca148c52949dc9e30caf2d91fe8cbdfd630f3e1e387b26f89a060648\": container with ID starting with 61b59d47ca148c52949dc9e30caf2d91fe8cbdfd630f3e1e387b26f89a060648 not found: ID does not exist" containerID="61b59d47ca148c52949dc9e30caf2d91fe8cbdfd630f3e1e387b26f89a060648" Nov 02 18:21:57.717887 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.717818 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b59d47ca148c52949dc9e30caf2d91fe8cbdfd630f3e1e387b26f89a060648"} err="failed to get container status \"61b59d47ca148c52949dc9e30caf2d91fe8cbdfd630f3e1e387b26f89a060648\": rpc error: code = NotFound desc = could not find container \"61b59d47ca148c52949dc9e30caf2d91fe8cbdfd630f3e1e387b26f89a060648\": container with ID starting with 61b59d47ca148c52949dc9e30caf2d91fe8cbdfd630f3e1e387b26f89a060648 not found: ID does not exist" Nov 02 18:21:57.717887 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.717835 5601 scope.go:122] "RemoveContainer" containerID="4f94841406dd48759f969b3f451b0570036eb289bfe818515e0b5e26a9451b3d" Nov 02 18:21:57.718176 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:21:57.718126 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f94841406dd48759f969b3f451b0570036eb289bfe818515e0b5e26a9451b3d\": container with ID starting with 4f94841406dd48759f969b3f451b0570036eb289bfe818515e0b5e26a9451b3d not found: ID does not exist" containerID="4f94841406dd48759f969b3f451b0570036eb289bfe818515e0b5e26a9451b3d" Nov 02 18:21:57.718176 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.718152 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f94841406dd48759f969b3f451b0570036eb289bfe818515e0b5e26a9451b3d"} err="failed to get container status \"4f94841406dd48759f969b3f451b0570036eb289bfe818515e0b5e26a9451b3d\": rpc error: code = NotFound desc = could not find container \"4f94841406dd48759f969b3f451b0570036eb289bfe818515e0b5e26a9451b3d\": container with ID starting with 4f94841406dd48759f969b3f451b0570036eb289bfe818515e0b5e26a9451b3d not found: ID does not exist" Nov 02 18:21:57.721920 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.721897 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz"] Nov 02 18:21:57.726198 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.726147 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-sr5cz"] Nov 02 18:21:57.747537 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.747510 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:57.747537 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.747536 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:57.747688 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.747546 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945a032d-c69f-4d9e-96a5-db1ac483af67-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:57.747688 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.747555 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjdck\" (UniqueName: \"kubernetes.io/projected/945a032d-c69f-4d9e-96a5-db1ac483af67-kube-api-access-zjdck\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:21:57.884681 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:57.884636 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" podUID="1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:21:58.566963 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:58.566924 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr"] Nov 02 18:21:59.429309 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:21:59.429274 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="945a032d-c69f-4d9e-96a5-db1ac483af67" path="/var/lib/kubelet/pods/945a032d-c69f-4d9e-96a5-db1ac483af67/volumes" Nov 02 18:22:03.568180 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:03.568133 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" podUID="ae604043-587b-4fd4-b923-4bd1b6170df2" containerName="payload-pre-processing" containerID="cri-o://ff8e22864822f427997c81cbd655fc40f2a702a78d3be76a0d6759b4a868542a" gracePeriod=25 Nov 02 18:22:04.486326 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:22:04.486285 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:22:07.884158 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:07.884061 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" podUID="1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:22:09.795140 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:09.795092 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" podUID="ae604043-587b-4fd4-b923-4bd1b6170df2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:22:10.199984 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:10.199956 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" Nov 02 18:22:10.228049 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:10.228009 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5-kube-api-access-pvqz6\" (UniqueName: \"kubernetes.io/projected/1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5-kube-api-access-pvqz6\") pod \"1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5\" (UID: \"1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5\") " Nov 02 18:22:10.228438 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:10.228410 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5-plugins-config-volume\") pod \"1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5\" (UID: \"1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5\") " Nov 02 18:22:10.228831 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:10.228809 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5-plugins-config-volume" pod "1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5" (UID: "1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:22:10.229111 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:10.229091 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:10.230321 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:10.230296 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5-kube-api-access-pvqz6" pod "1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5" (UID: "1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5"). InnerVolumeSpecName "kube-api-access-pvqz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:22:10.330676 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:10.330637 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvqz6\" (UniqueName: \"kubernetes.io/projected/1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5-kube-api-access-pvqz6\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:10.809457 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:10.809411 5601 generic.go:356] "Generic (PLEG): container finished" podID="1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5" containerID="ca796f7acc8a4f956cc40f86d3bcbf388a1b9ec2f56c6a2d5cee601c41f64bd8" exitCode=0 Nov 02 18:22:10.810042 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:10.809476 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" Nov 02 18:22:10.810042 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:10.809495 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" event={"ID":"1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5","Type":"ContainerDied","Data":"ca796f7acc8a4f956cc40f86d3bcbf388a1b9ec2f56c6a2d5cee601c41f64bd8"} Nov 02 18:22:10.810042 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:10.809551 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr" event={"ID":"1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5","Type":"ContainerDied","Data":"df3ff91b18e6bb47771f6859e1fa73e436da39804dd1f5f571196bf64a40c8cb"} Nov 02 18:22:10.810042 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:10.809574 5601 scope.go:122] "RemoveContainer" containerID="ca796f7acc8a4f956cc40f86d3bcbf388a1b9ec2f56c6a2d5cee601c41f64bd8" Nov 02 18:22:10.821747 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:10.821724 5601 scope.go:122] "RemoveContainer" containerID="ca796f7acc8a4f956cc40f86d3bcbf388a1b9ec2f56c6a2d5cee601c41f64bd8" Nov 02 18:22:10.822049 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:22:10.822029 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca796f7acc8a4f956cc40f86d3bcbf388a1b9ec2f56c6a2d5cee601c41f64bd8\": container with ID starting with ca796f7acc8a4f956cc40f86d3bcbf388a1b9ec2f56c6a2d5cee601c41f64bd8 not found: ID does not exist" containerID="ca796f7acc8a4f956cc40f86d3bcbf388a1b9ec2f56c6a2d5cee601c41f64bd8" Nov 02 18:22:10.822108 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:10.822056 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca796f7acc8a4f956cc40f86d3bcbf388a1b9ec2f56c6a2d5cee601c41f64bd8"} err="failed to get container status \"ca796f7acc8a4f956cc40f86d3bcbf388a1b9ec2f56c6a2d5cee601c41f64bd8\": rpc error: code = NotFound desc = could not find container \"ca796f7acc8a4f956cc40f86d3bcbf388a1b9ec2f56c6a2d5cee601c41f64bd8\": container with ID starting with ca796f7acc8a4f956cc40f86d3bcbf388a1b9ec2f56c6a2d5cee601c41f64bd8 not found: ID does not exist" Nov 02 18:22:10.839452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:10.839424 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr"] Nov 02 18:22:10.845545 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:10.845520 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-312f21-56cc99d475-qtdtr"] Nov 02 18:22:11.430368 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:11.430326 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5" path="/var/lib/kubelet/pods/1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5/volumes" Nov 02 18:22:12.721666 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:12.717911 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg"] Nov 02 18:22:12.832653 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:12.832584 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" podUID="5717d304-fd02-42a4-a04f-0958ce43fdb0" containerName="maas-api" containerID="cri-o://bac52e3a96202835bce66d2579634249e819254f62177d70d266894153095460" gracePeriod=30 Nov 02 18:22:13.482794 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.482763 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" Nov 02 18:22:13.554794 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.554762 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5717d304-fd02-42a4-a04f-0958ce43fdb0-kube-api-access-56fwp\" (UniqueName: \"kubernetes.io/projected/5717d304-fd02-42a4-a04f-0958ce43fdb0-kube-api-access-56fwp\") pod \"5717d304-fd02-42a4-a04f-0958ce43fdb0\" (UID: \"5717d304-fd02-42a4-a04f-0958ce43fdb0\") " Nov 02 18:22:13.554973 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.554896 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5717d304-fd02-42a4-a04f-0958ce43fdb0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5717d304-fd02-42a4-a04f-0958ce43fdb0-metrics-certs\") pod \"5717d304-fd02-42a4-a04f-0958ce43fdb0\" (UID: \"5717d304-fd02-42a4-a04f-0958ce43fdb0\") " Nov 02 18:22:13.555054 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.555020 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5717d304-fd02-42a4-a04f-0958ce43fdb0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5717d304-fd02-42a4-a04f-0958ce43fdb0-maas-api-tls\") pod \"5717d304-fd02-42a4-a04f-0958ce43fdb0\" (UID: \"5717d304-fd02-42a4-a04f-0958ce43fdb0\") " Nov 02 18:22:13.555135 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.555119 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5717d304-fd02-42a4-a04f-0958ce43fdb0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5717d304-fd02-42a4-a04f-0958ce43fdb0-openshift-service-ca\") pod \"5717d304-fd02-42a4-a04f-0958ce43fdb0\" (UID: \"5717d304-fd02-42a4-a04f-0958ce43fdb0\") " Nov 02 18:22:13.556951 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.556922 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5717d304-fd02-42a4-a04f-0958ce43fdb0-openshift-service-ca" pod "5717d304-fd02-42a4-a04f-0958ce43fdb0" (UID: "5717d304-fd02-42a4-a04f-0958ce43fdb0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:22:13.557574 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.557546 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5717d304-fd02-42a4-a04f-0958ce43fdb0-metrics-certs" pod "5717d304-fd02-42a4-a04f-0958ce43fdb0" (UID: "5717d304-fd02-42a4-a04f-0958ce43fdb0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:22:13.557709 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.557610 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5717d304-fd02-42a4-a04f-0958ce43fdb0-kube-api-access-56fwp" pod "5717d304-fd02-42a4-a04f-0958ce43fdb0" (UID: "5717d304-fd02-42a4-a04f-0958ce43fdb0"). InnerVolumeSpecName "kube-api-access-56fwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:22:13.558246 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.558225 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5717d304-fd02-42a4-a04f-0958ce43fdb0-maas-api-tls" pod "5717d304-fd02-42a4-a04f-0958ce43fdb0" (UID: "5717d304-fd02-42a4-a04f-0958ce43fdb0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:22:13.657960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.657886 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5717d304-fd02-42a4-a04f-0958ce43fdb0-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:13.657960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.657916 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56fwp\" (UniqueName: \"kubernetes.io/projected/5717d304-fd02-42a4-a04f-0958ce43fdb0-kube-api-access-56fwp\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:13.657960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.657932 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5717d304-fd02-42a4-a04f-0958ce43fdb0-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:13.657960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.657942 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5717d304-fd02-42a4-a04f-0958ce43fdb0-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:13.842174 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.842139 5601 generic.go:356] "Generic (PLEG): container finished" podID="5717d304-fd02-42a4-a04f-0958ce43fdb0" containerID="bac52e3a96202835bce66d2579634249e819254f62177d70d266894153095460" exitCode=0 Nov 02 18:22:13.842641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.842198 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" Nov 02 18:22:13.842641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.842225 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" event={"ID":"5717d304-fd02-42a4-a04f-0958ce43fdb0","Type":"ContainerDied","Data":"bac52e3a96202835bce66d2579634249e819254f62177d70d266894153095460"} Nov 02 18:22:13.842641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.842289 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg" event={"ID":"5717d304-fd02-42a4-a04f-0958ce43fdb0","Type":"ContainerDied","Data":"c929077793035509f326b6934d1e67f2a95a6e33310bde3703fa461028d3f7ca"} Nov 02 18:22:13.842641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.842314 5601 scope.go:122] "RemoveContainer" containerID="bac52e3a96202835bce66d2579634249e819254f62177d70d266894153095460" Nov 02 18:22:13.855639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.855606 5601 scope.go:122] "RemoveContainer" containerID="bac52e3a96202835bce66d2579634249e819254f62177d70d266894153095460" Nov 02 18:22:13.856107 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:22:13.856074 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac52e3a96202835bce66d2579634249e819254f62177d70d266894153095460\": container with ID starting with bac52e3a96202835bce66d2579634249e819254f62177d70d266894153095460 not found: ID does not exist" containerID="bac52e3a96202835bce66d2579634249e819254f62177d70d266894153095460" Nov 02 18:22:13.856248 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.856117 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac52e3a96202835bce66d2579634249e819254f62177d70d266894153095460"} err="failed to get container status \"bac52e3a96202835bce66d2579634249e819254f62177d70d266894153095460\": rpc error: code = NotFound desc = could not find container \"bac52e3a96202835bce66d2579634249e819254f62177d70d266894153095460\": container with ID starting with bac52e3a96202835bce66d2579634249e819254f62177d70d266894153095460 not found: ID does not exist" Nov 02 18:22:13.874855 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.874827 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg"] Nov 02 18:22:13.881932 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:13.881907 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0610-6b94dd7c7b-n2hmg"] Nov 02 18:22:15.428628 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:15.428576 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5717d304-fd02-42a4-a04f-0958ce43fdb0" path="/var/lib/kubelet/pods/5717d304-fd02-42a4-a04f-0958ce43fdb0/volumes" Nov 02 18:22:19.795238 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:19.795199 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" podUID="ae604043-587b-4fd4-b923-4bd1b6170df2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:22:22.588332 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:22.588284 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k"] Nov 02 18:22:22.588916 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:22.588570 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" podUID="5c2aa306-9a80-4299-9ada-d7229e813dee" containerName="istio-proxy" containerID="cri-o://c09fba55cae07e47a4831733ab1d8cb7d959a4b8a25dd20d60574f366aff63a4" gracePeriod=30 Nov 02 18:22:23.943452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:23.943390 5601 generic.go:356] "Generic (PLEG): container finished" podID="ae604043-587b-4fd4-b923-4bd1b6170df2" containerID="ff8e22864822f427997c81cbd655fc40f2a702a78d3be76a0d6759b4a868542a" exitCode=0 Nov 02 18:22:23.943957 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:23.943806 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" event={"ID":"ae604043-587b-4fd4-b923-4bd1b6170df2","Type":"ContainerDied","Data":"ff8e22864822f427997c81cbd655fc40f2a702a78d3be76a0d6759b4a868542a"} Nov 02 18:22:24.017412 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:24.017385 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" Nov 02 18:22:24.148739 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:24.148649 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae604043-587b-4fd4-b923-4bd1b6170df2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae604043-587b-4fd4-b923-4bd1b6170df2-plugins-config-volume\") pod \"ae604043-587b-4fd4-b923-4bd1b6170df2\" (UID: \"ae604043-587b-4fd4-b923-4bd1b6170df2\") " Nov 02 18:22:24.148919 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:24.148848 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae604043-587b-4fd4-b923-4bd1b6170df2-kube-api-access-g4fvw\" (UniqueName: \"kubernetes.io/projected/ae604043-587b-4fd4-b923-4bd1b6170df2-kube-api-access-g4fvw\") pod \"ae604043-587b-4fd4-b923-4bd1b6170df2\" (UID: \"ae604043-587b-4fd4-b923-4bd1b6170df2\") " Nov 02 18:22:24.149115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:24.149086 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae604043-587b-4fd4-b923-4bd1b6170df2-plugins-config-volume" pod "ae604043-587b-4fd4-b923-4bd1b6170df2" (UID: "ae604043-587b-4fd4-b923-4bd1b6170df2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:22:24.149756 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:24.149727 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae604043-587b-4fd4-b923-4bd1b6170df2-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:24.151283 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:24.151253 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae604043-587b-4fd4-b923-4bd1b6170df2-kube-api-access-g4fvw" pod "ae604043-587b-4fd4-b923-4bd1b6170df2" (UID: "ae604043-587b-4fd4-b923-4bd1b6170df2"). InnerVolumeSpecName "kube-api-access-g4fvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:22:24.251446 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:24.251397 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4fvw\" (UniqueName: \"kubernetes.io/projected/ae604043-587b-4fd4-b923-4bd1b6170df2-kube-api-access-g4fvw\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:24.955429 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:24.955392 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" event={"ID":"ae604043-587b-4fd4-b923-4bd1b6170df2","Type":"ContainerDied","Data":"4285ce2309eecea08899bdca95e334e46f60be478132a43b56e131e054440e69"} Nov 02 18:22:24.955429 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:24.955417 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr" Nov 02 18:22:24.956023 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:24.955478 5601 scope.go:122] "RemoveContainer" containerID="ff8e22864822f427997c81cbd655fc40f2a702a78d3be76a0d6759b4a868542a" Nov 02 18:22:24.983711 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:24.983684 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr"] Nov 02 18:22:24.987516 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:24.987495 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0b0610-7d957d5bb7-thwqr"] Nov 02 18:22:25.445047 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:25.445001 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae604043-587b-4fd4-b923-4bd1b6170df2" path="/var/lib/kubelet/pods/ae604043-587b-4fd4-b923-4bd1b6170df2/volumes" Nov 02 18:22:27.896814 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.896744 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:22:27.937315 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.937278 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c2aa306-9a80-4299-9ada-d7229e813dee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2aa306-9a80-4299-9ada-d7229e813dee-istiod-ca-cert\") pod \"5c2aa306-9a80-4299-9ada-d7229e813dee\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " Nov 02 18:22:27.937517 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.937395 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-credential-socket\") pod \"5c2aa306-9a80-4299-9ada-d7229e813dee\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " Nov 02 18:22:27.937517 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.937453 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-workload-certs\") pod \"5c2aa306-9a80-4299-9ada-d7229e813dee\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " Nov 02 18:22:27.937708 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.937534 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2aa306-9a80-4299-9ada-d7229e813dee-kube-api-access-tck28\" (UniqueName: \"kubernetes.io/projected/5c2aa306-9a80-4299-9ada-d7229e813dee-kube-api-access-tck28\") pod \"5c2aa306-9a80-4299-9ada-d7229e813dee\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " Nov 02 18:22:27.937708 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.937657 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-envoy\") pod \"5c2aa306-9a80-4299-9ada-d7229e813dee\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " Nov 02 18:22:27.937708 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.937702 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-data\") pod \"5c2aa306-9a80-4299-9ada-d7229e813dee\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " Nov 02 18:22:27.937879 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.937756 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-workload-socket\") pod \"5c2aa306-9a80-4299-9ada-d7229e813dee\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " Nov 02 18:22:27.937879 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.937768 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2aa306-9a80-4299-9ada-d7229e813dee-istiod-ca-cert" pod "5c2aa306-9a80-4299-9ada-d7229e813dee" (UID: "5c2aa306-9a80-4299-9ada-d7229e813dee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:22:27.937986 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.937896 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-podinfo\") pod \"5c2aa306-9a80-4299-9ada-d7229e813dee\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " Nov 02 18:22:27.938043 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.938009 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-workload-socket" pod "5c2aa306-9a80-4299-9ada-d7229e813dee" (UID: "5c2aa306-9a80-4299-9ada-d7229e813dee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:22:27.938106 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.938093 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-token\" (UniqueName: \"kubernetes.io/projected/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-token\") pod \"5c2aa306-9a80-4299-9ada-d7229e813dee\" (UID: \"5c2aa306-9a80-4299-9ada-d7229e813dee\") " Nov 02 18:22:27.938165 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.938147 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-workload-certs" pod "5c2aa306-9a80-4299-9ada-d7229e813dee" (UID: "5c2aa306-9a80-4299-9ada-d7229e813dee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:22:27.938275 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.938247 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-data" pod "5c2aa306-9a80-4299-9ada-d7229e813dee" (UID: "5c2aa306-9a80-4299-9ada-d7229e813dee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:22:27.938551 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.938518 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-credential-socket" pod "5c2aa306-9a80-4299-9ada-d7229e813dee" (UID: "5c2aa306-9a80-4299-9ada-d7229e813dee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:22:27.938981 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.938955 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:27.939084 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.938986 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:27.939084 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.939004 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2aa306-9a80-4299-9ada-d7229e813dee-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:27.939084 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.939020 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:27.939084 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.939035 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:27.940429 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.940403 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-envoy" pod "5c2aa306-9a80-4299-9ada-d7229e813dee" (UID: "5c2aa306-9a80-4299-9ada-d7229e813dee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:22:27.940660 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.940637 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-podinfo" pod "5c2aa306-9a80-4299-9ada-d7229e813dee" (UID: "5c2aa306-9a80-4299-9ada-d7229e813dee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:22:27.940855 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.940828 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-token" pod "5c2aa306-9a80-4299-9ada-d7229e813dee" (UID: "5c2aa306-9a80-4299-9ada-d7229e813dee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:22:27.946055 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.946029 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2aa306-9a80-4299-9ada-d7229e813dee-kube-api-access-tck28" pod "5c2aa306-9a80-4299-9ada-d7229e813dee" (UID: "5c2aa306-9a80-4299-9ada-d7229e813dee"). InnerVolumeSpecName "kube-api-access-tck28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:22:27.984983 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.984944 5601 generic.go:356] "Generic (PLEG): container finished" podID="5c2aa306-9a80-4299-9ada-d7229e813dee" containerID="c09fba55cae07e47a4831733ab1d8cb7d959a4b8a25dd20d60574f366aff63a4" exitCode=0 Nov 02 18:22:27.985186 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.985004 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" Nov 02 18:22:27.985186 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.985183 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" event={"ID":"5c2aa306-9a80-4299-9ada-d7229e813dee","Type":"ContainerDied","Data":"c09fba55cae07e47a4831733ab1d8cb7d959a4b8a25dd20d60574f366aff63a4"} Nov 02 18:22:27.985343 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.985224 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k" event={"ID":"5c2aa306-9a80-4299-9ada-d7229e813dee","Type":"ContainerDied","Data":"462f9af4aa927b1d0fa7e42af1e096419635b06b1ffb3f54854a8b64470a4b4f"} Nov 02 18:22:27.985343 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:27.985246 5601 scope.go:122] "RemoveContainer" containerID="c09fba55cae07e47a4831733ab1d8cb7d959a4b8a25dd20d60574f366aff63a4" Nov 02 18:22:28.001197 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:28.001170 5601 scope.go:122] "RemoveContainer" containerID="e931d5cf9949b1434fbe01a656b3bbc428c5257df06f89676106190f88818a29" Nov 02 18:22:28.016006 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:28.015983 5601 scope.go:122] "RemoveContainer" containerID="c09fba55cae07e47a4831733ab1d8cb7d959a4b8a25dd20d60574f366aff63a4" Nov 02 18:22:28.016348 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:22:28.016327 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c09fba55cae07e47a4831733ab1d8cb7d959a4b8a25dd20d60574f366aff63a4\": container with ID starting with c09fba55cae07e47a4831733ab1d8cb7d959a4b8a25dd20d60574f366aff63a4 not found: ID does not exist" containerID="c09fba55cae07e47a4831733ab1d8cb7d959a4b8a25dd20d60574f366aff63a4" Nov 02 18:22:28.016457 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:28.016358 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c09fba55cae07e47a4831733ab1d8cb7d959a4b8a25dd20d60574f366aff63a4"} err="failed to get container status \"c09fba55cae07e47a4831733ab1d8cb7d959a4b8a25dd20d60574f366aff63a4\": rpc error: code = NotFound desc = could not find container \"c09fba55cae07e47a4831733ab1d8cb7d959a4b8a25dd20d60574f366aff63a4\": container with ID starting with c09fba55cae07e47a4831733ab1d8cb7d959a4b8a25dd20d60574f366aff63a4 not found: ID does not exist" Nov 02 18:22:28.016457 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:28.016383 5601 scope.go:122] "RemoveContainer" containerID="e931d5cf9949b1434fbe01a656b3bbc428c5257df06f89676106190f88818a29" Nov 02 18:22:28.016688 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:22:28.016658 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e931d5cf9949b1434fbe01a656b3bbc428c5257df06f89676106190f88818a29\": container with ID starting with e931d5cf9949b1434fbe01a656b3bbc428c5257df06f89676106190f88818a29 not found: ID does not exist" containerID="e931d5cf9949b1434fbe01a656b3bbc428c5257df06f89676106190f88818a29" Nov 02 18:22:28.016768 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:28.016699 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e931d5cf9949b1434fbe01a656b3bbc428c5257df06f89676106190f88818a29"} err="failed to get container status \"e931d5cf9949b1434fbe01a656b3bbc428c5257df06f89676106190f88818a29\": rpc error: code = NotFound desc = could not find container \"e931d5cf9949b1434fbe01a656b3bbc428c5257df06f89676106190f88818a29\": container with ID starting with e931d5cf9949b1434fbe01a656b3bbc428c5257df06f89676106190f88818a29 not found: ID does not exist" Nov 02 18:22:28.018983 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:28.018958 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k"] Nov 02 18:22:28.030201 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:28.029948 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0b0610-openshift-default-6df9db6447-pdd2k"] Nov 02 18:22:28.041020 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:28.040988 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:28.041020 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:28.041018 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tck28\" (UniqueName: \"kubernetes.io/projected/5c2aa306-9a80-4299-9ada-d7229e813dee-kube-api-access-tck28\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:28.041204 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:28.041030 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:28.041204 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:28.041039 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2aa306-9a80-4299-9ada-d7229e813dee-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:29.430208 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:29.430172 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c2aa306-9a80-4299-9ada-d7229e813dee" path="/var/lib/kubelet/pods/5c2aa306-9a80-4299-9ada-d7229e813dee/volumes" Nov 02 18:22:32.140468 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:32.140427 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz"] Nov 02 18:22:33.384287 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:33.384253 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj"] Nov 02 18:22:33.385196 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:33.385165 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" podUID="196bcedf-fdfd-4469-9234-d281f907d53a" containerName="istio-proxy" containerID="cri-o://a61f531e9ab0c910390c112acf6a8bf15a30f53e37c5ffabf070b84bdb9a8954" gracePeriod=30 Nov 02 18:22:34.595528 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:22:34.595443 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:22:38.226472 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:38.226427 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:22:38.227017 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:38.226507 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" podUID="196bcedf-fdfd-4469-9234-d281f907d53a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:22:38.232331 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:38.232296 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5"] Nov 02 18:22:38.952081 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:38.952048 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 18:22:39.138646 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.138528 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-credential-socket\") pod \"196bcedf-fdfd-4469-9234-d281f907d53a\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " Nov 02 18:22:39.138950 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.138927 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/196bcedf-fdfd-4469-9234-d281f907d53a-istio-token\" (UniqueName: \"kubernetes.io/projected/196bcedf-fdfd-4469-9234-d281f907d53a-istio-token\") pod \"196bcedf-fdfd-4469-9234-d281f907d53a\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " Nov 02 18:22:39.145846 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.145810 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-istio-data\") pod \"196bcedf-fdfd-4469-9234-d281f907d53a\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " Nov 02 18:22:39.170045 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.170014 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/196bcedf-fdfd-4469-9234-d281f907d53a-kube-api-access-stx8m\" (UniqueName: \"kubernetes.io/projected/196bcedf-fdfd-4469-9234-d281f907d53a-kube-api-access-stx8m\") pod \"196bcedf-fdfd-4469-9234-d281f907d53a\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " Nov 02 18:22:39.170263 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.170246 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-workload-socket\") pod \"196bcedf-fdfd-4469-9234-d281f907d53a\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " Nov 02 18:22:39.197641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.170434 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/196bcedf-fdfd-4469-9234-d281f907d53a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196bcedf-fdfd-4469-9234-d281f907d53a-istio-podinfo\") pod \"196bcedf-fdfd-4469-9234-d281f907d53a\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " Nov 02 18:22:39.198001 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.197972 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-istio-envoy\") pod \"196bcedf-fdfd-4469-9234-d281f907d53a\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " Nov 02 18:22:39.198128 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.198113 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-workload-certs\") pod \"196bcedf-fdfd-4469-9234-d281f907d53a\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " Nov 02 18:22:39.198248 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.198234 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/196bcedf-fdfd-4469-9234-d281f907d53a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196bcedf-fdfd-4469-9234-d281f907d53a-istiod-ca-cert\") pod \"196bcedf-fdfd-4469-9234-d281f907d53a\" (UID: \"196bcedf-fdfd-4469-9234-d281f907d53a\") " Nov 02 18:22:39.206788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.146176 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-credential-socket" pod "196bcedf-fdfd-4469-9234-d281f907d53a" (UID: "196bcedf-fdfd-4469-9234-d281f907d53a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:22:39.206788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.174435 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196bcedf-fdfd-4469-9234-d281f907d53a-istio-token" pod "196bcedf-fdfd-4469-9234-d281f907d53a" (UID: "196bcedf-fdfd-4469-9234-d281f907d53a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:22:39.206788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.174817 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-istio-data" pod "196bcedf-fdfd-4469-9234-d281f907d53a" (UID: "196bcedf-fdfd-4469-9234-d281f907d53a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:22:39.206788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.183542 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-workload-socket" pod "196bcedf-fdfd-4469-9234-d281f907d53a" (UID: "196bcedf-fdfd-4469-9234-d281f907d53a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:22:39.206788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.204059 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196bcedf-fdfd-4469-9234-d281f907d53a-kube-api-access-stx8m" pod "196bcedf-fdfd-4469-9234-d281f907d53a" (UID: "196bcedf-fdfd-4469-9234-d281f907d53a"). InnerVolumeSpecName "kube-api-access-stx8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:22:39.206788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.204304 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-workload-certs" pod "196bcedf-fdfd-4469-9234-d281f907d53a" (UID: "196bcedf-fdfd-4469-9234-d281f907d53a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:22:39.206788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.203940 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/196bcedf-fdfd-4469-9234-d281f907d53a-istio-podinfo" pod "196bcedf-fdfd-4469-9234-d281f907d53a" (UID: "196bcedf-fdfd-4469-9234-d281f907d53a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:22:39.206788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.205922 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/196bcedf-fdfd-4469-9234-d281f907d53a-istiod-ca-cert" pod "196bcedf-fdfd-4469-9234-d281f907d53a" (UID: "196bcedf-fdfd-4469-9234-d281f907d53a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:22:39.206788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.206607 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:39.206788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.206651 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/196bcedf-fdfd-4469-9234-d281f907d53a-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:39.206788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.206665 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:39.206788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.206680 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stx8m\" (UniqueName: \"kubernetes.io/projected/196bcedf-fdfd-4469-9234-d281f907d53a-kube-api-access-stx8m\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:39.206788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.206696 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:39.206788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.206709 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/196bcedf-fdfd-4469-9234-d281f907d53a-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:39.206788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.206723 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:39.206788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.206736 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/196bcedf-fdfd-4469-9234-d281f907d53a-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:39.213106 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.213071 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-istio-envoy" pod "196bcedf-fdfd-4469-9234-d281f907d53a" (UID: "196bcedf-fdfd-4469-9234-d281f907d53a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:22:39.239378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.237263 5601 generic.go:356] "Generic (PLEG): container finished" podID="196bcedf-fdfd-4469-9234-d281f907d53a" containerID="a61f531e9ab0c910390c112acf6a8bf15a30f53e37c5ffabf070b84bdb9a8954" exitCode=0 Nov 02 18:22:39.239378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.237530 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" event={"ID":"196bcedf-fdfd-4469-9234-d281f907d53a","Type":"ContainerDied","Data":"a61f531e9ab0c910390c112acf6a8bf15a30f53e37c5ffabf070b84bdb9a8954"} Nov 02 18:22:39.239378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.237590 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" event={"ID":"196bcedf-fdfd-4469-9234-d281f907d53a","Type":"ContainerDied","Data":"8334079d2c4a8b8e49d059f530a720b5fe64586a32f37d0c89f8bb67b9550d45"} Nov 02 18:22:39.239378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.237716 5601 scope.go:122] "RemoveContainer" containerID="a61f531e9ab0c910390c112acf6a8bf15a30f53e37c5ffabf070b84bdb9a8954" Nov 02 18:22:39.239378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.237892 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj" Nov 02 18:22:39.265591 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.265569 5601 scope.go:122] "RemoveContainer" containerID="a61f531e9ab0c910390c112acf6a8bf15a30f53e37c5ffabf070b84bdb9a8954" Nov 02 18:22:39.266052 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:22:39.266005 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61f531e9ab0c910390c112acf6a8bf15a30f53e37c5ffabf070b84bdb9a8954\": container with ID starting with a61f531e9ab0c910390c112acf6a8bf15a30f53e37c5ffabf070b84bdb9a8954 not found: ID does not exist" containerID="a61f531e9ab0c910390c112acf6a8bf15a30f53e37c5ffabf070b84bdb9a8954" Nov 02 18:22:39.266158 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.266065 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61f531e9ab0c910390c112acf6a8bf15a30f53e37c5ffabf070b84bdb9a8954"} err="failed to get container status \"a61f531e9ab0c910390c112acf6a8bf15a30f53e37c5ffabf070b84bdb9a8954\": rpc error: code = NotFound desc = could not find container \"a61f531e9ab0c910390c112acf6a8bf15a30f53e37c5ffabf070b84bdb9a8954\": container with ID starting with a61f531e9ab0c910390c112acf6a8bf15a30f53e37c5ffabf070b84bdb9a8954 not found: ID does not exist" Nov 02 18:22:39.285790 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.285730 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj"] Nov 02 18:22:39.292482 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.292451 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-89qtj"] Nov 02 18:22:39.308549 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.308523 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/196bcedf-fdfd-4469-9234-d281f907d53a-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:39.430449 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:39.430411 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="196bcedf-fdfd-4469-9234-d281f907d53a" path="/var/lib/kubelet/pods/196bcedf-fdfd-4469-9234-d281f907d53a/volumes" Nov 02 18:22:43.233197 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:43.233134 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" podUID="64765539-809c-42a0-af1c-7d1e6b506b18" containerName="payload-pre-processing" containerID="cri-o://1090fa61dc0cbe07ede96e593d5cdeb79c8b0dbc9b256f55a24fbf0840657ef5" gracePeriod=25 Nov 02 18:22:45.137970 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:45.137927 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" podUID="64765539-809c-42a0-af1c-7d1e6b506b18" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:22:47.166348 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.166295 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" podUID="ab48f110-7cc1-45fb-a416-527d9c919917" containerName="main" containerID="cri-o://6072c6b790ec75e21fa28a40f911d33b75fcc92421d3dfa2b167d3985c3c9e21" gracePeriod=45 Nov 02 18:22:47.287145 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.287101 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" podUID="ab48f110-7cc1-45fb-a416-527d9c919917" containerName="main" probeResult="failure" output="Get \"https://10.134.0.187:8000/ready\": dial tcp 10.134.0.187:8000: connect: connection refused" Nov 02 18:22:47.308592 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.308553 5601 generic.go:356] "Generic (PLEG): container finished" podID="ab48f110-7cc1-45fb-a416-527d9c919917" containerID="6072c6b790ec75e21fa28a40f911d33b75fcc92421d3dfa2b167d3985c3c9e21" exitCode=0 Nov 02 18:22:47.308786 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.308601 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" event={"ID":"ab48f110-7cc1-45fb-a416-527d9c919917","Type":"ContainerDied","Data":"6072c6b790ec75e21fa28a40f911d33b75fcc92421d3dfa2b167d3985c3c9e21"} Nov 02 18:22:47.481564 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.481521 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:22:47.550273 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.548760 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-model-cache\") pod \"ab48f110-7cc1-45fb-a416-527d9c919917\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " Nov 02 18:22:47.550273 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.548806 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-dshm\") pod \"ab48f110-7cc1-45fb-a416-527d9c919917\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " Nov 02 18:22:47.550273 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.548882 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab48f110-7cc1-45fb-a416-527d9c919917-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab48f110-7cc1-45fb-a416-527d9c919917-tls-certs\") pod \"ab48f110-7cc1-45fb-a416-527d9c919917\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " Nov 02 18:22:47.550273 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.549067 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-model-cache" pod "ab48f110-7cc1-45fb-a416-527d9c919917" (UID: "ab48f110-7cc1-45fb-a416-527d9c919917"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:22:47.550273 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.549086 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab48f110-7cc1-45fb-a416-527d9c919917-kube-api-access-n2db6\" (UniqueName: \"kubernetes.io/projected/ab48f110-7cc1-45fb-a416-527d9c919917-kube-api-access-n2db6\") pod \"ab48f110-7cc1-45fb-a416-527d9c919917\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " Nov 02 18:22:47.550273 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.549245 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-tmp-dir\") pod \"ab48f110-7cc1-45fb-a416-527d9c919917\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " Nov 02 18:22:47.550273 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.549297 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-home\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-home\") pod \"ab48f110-7cc1-45fb-a416-527d9c919917\" (UID: \"ab48f110-7cc1-45fb-a416-527d9c919917\") " Nov 02 18:22:47.550273 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.550050 5601 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-model-cache\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:47.551525 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.551415 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-tmp-dir" pod "ab48f110-7cc1-45fb-a416-527d9c919917" (UID: "ab48f110-7cc1-45fb-a416-527d9c919917"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:22:47.551804 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.551778 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-home" pod "ab48f110-7cc1-45fb-a416-527d9c919917" (UID: "ab48f110-7cc1-45fb-a416-527d9c919917"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:22:47.552260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.552229 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab48f110-7cc1-45fb-a416-527d9c919917-tls-certs" pod "ab48f110-7cc1-45fb-a416-527d9c919917" (UID: "ab48f110-7cc1-45fb-a416-527d9c919917"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:22:47.560240 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.560215 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab48f110-7cc1-45fb-a416-527d9c919917-kube-api-access-n2db6" pod "ab48f110-7cc1-45fb-a416-527d9c919917" (UID: "ab48f110-7cc1-45fb-a416-527d9c919917"). InnerVolumeSpecName "kube-api-access-n2db6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:22:47.565314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.565291 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-dshm" pod "ab48f110-7cc1-45fb-a416-527d9c919917" (UID: "ab48f110-7cc1-45fb-a416-527d9c919917"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:22:47.651308 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.651263 5601 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab48f110-7cc1-45fb-a416-527d9c919917-tls-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:47.651308 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.651300 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2db6\" (UniqueName: \"kubernetes.io/projected/ab48f110-7cc1-45fb-a416-527d9c919917-kube-api-access-n2db6\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:47.651308 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.651311 5601 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-tmp-dir\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:47.651587 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.651321 5601 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-home\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:47.651587 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:47.651329 5601 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab48f110-7cc1-45fb-a416-527d9c919917-dshm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:48.318224 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:48.318177 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" event={"ID":"ab48f110-7cc1-45fb-a416-527d9c919917","Type":"ContainerDied","Data":"2578802dfad3764d3b397ceb120047d315abca111b35c3c27ffe1e465f13832d"} Nov 02 18:22:48.318224 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:48.318211 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz" Nov 02 18:22:48.318787 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:48.318243 5601 scope.go:122] "RemoveContainer" containerID="6072c6b790ec75e21fa28a40f911d33b75fcc92421d3dfa2b167d3985c3c9e21" Nov 02 18:22:48.344863 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:48.344830 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz"] Nov 02 18:22:48.348885 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:48.348858 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-63be7a/test-model-63be7a-kserve-7bbf984f67-sp5cz"] Nov 02 18:22:49.428946 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:49.428912 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab48f110-7cc1-45fb-a416-527d9c919917" path="/var/lib/kubelet/pods/ab48f110-7cc1-45fb-a416-527d9c919917/volumes" Nov 02 18:22:53.043567 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:53.043526 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch"] Nov 02 18:22:53.044127 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:53.043824 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" podUID="851b2b14-e25a-47b9-8eab-0e68096e6268" containerName="maas-api" containerID="cri-o://79a03d67e4d2a8e53d948bf06497af0aebcb690ad72d041bfbc0ac55b26a9ce8" gracePeriod=30 Nov 02 18:22:53.619939 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:53.619854 5601 generic.go:356] "Generic (PLEG): container finished" podID="851b2b14-e25a-47b9-8eab-0e68096e6268" containerID="79a03d67e4d2a8e53d948bf06497af0aebcb690ad72d041bfbc0ac55b26a9ce8" exitCode=0 Nov 02 18:22:53.619939 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:53.619929 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" event={"ID":"851b2b14-e25a-47b9-8eab-0e68096e6268","Type":"ContainerDied","Data":"79a03d67e4d2a8e53d948bf06497af0aebcb690ad72d041bfbc0ac55b26a9ce8"} Nov 02 18:22:53.699825 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:53.699799 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" Nov 02 18:22:53.884926 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:53.884838 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/851b2b14-e25a-47b9-8eab-0e68096e6268-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/851b2b14-e25a-47b9-8eab-0e68096e6268-maas-api-tls\") pod \"851b2b14-e25a-47b9-8eab-0e68096e6268\" (UID: \"851b2b14-e25a-47b9-8eab-0e68096e6268\") " Nov 02 18:22:53.885107 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:53.884955 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/851b2b14-e25a-47b9-8eab-0e68096e6268-kube-api-access-nh7jv\" (UniqueName: \"kubernetes.io/projected/851b2b14-e25a-47b9-8eab-0e68096e6268-kube-api-access-nh7jv\") pod \"851b2b14-e25a-47b9-8eab-0e68096e6268\" (UID: \"851b2b14-e25a-47b9-8eab-0e68096e6268\") " Nov 02 18:22:53.885107 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:53.885053 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/851b2b14-e25a-47b9-8eab-0e68096e6268-metrics-certs\" (UniqueName: \"kubernetes.io/secret/851b2b14-e25a-47b9-8eab-0e68096e6268-metrics-certs\") pod \"851b2b14-e25a-47b9-8eab-0e68096e6268\" (UID: \"851b2b14-e25a-47b9-8eab-0e68096e6268\") " Nov 02 18:22:53.885107 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:53.885090 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/851b2b14-e25a-47b9-8eab-0e68096e6268-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/851b2b14-e25a-47b9-8eab-0e68096e6268-openshift-service-ca\") pod \"851b2b14-e25a-47b9-8eab-0e68096e6268\" (UID: \"851b2b14-e25a-47b9-8eab-0e68096e6268\") " Nov 02 18:22:53.885473 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:53.885442 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/851b2b14-e25a-47b9-8eab-0e68096e6268-openshift-service-ca" pod "851b2b14-e25a-47b9-8eab-0e68096e6268" (UID: "851b2b14-e25a-47b9-8eab-0e68096e6268"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:22:53.885861 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:53.885841 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/851b2b14-e25a-47b9-8eab-0e68096e6268-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:53.887217 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:53.887159 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/851b2b14-e25a-47b9-8eab-0e68096e6268-maas-api-tls" pod "851b2b14-e25a-47b9-8eab-0e68096e6268" (UID: "851b2b14-e25a-47b9-8eab-0e68096e6268"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:22:53.887323 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:53.887290 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/851b2b14-e25a-47b9-8eab-0e68096e6268-metrics-certs" pod "851b2b14-e25a-47b9-8eab-0e68096e6268" (UID: "851b2b14-e25a-47b9-8eab-0e68096e6268"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:22:53.887448 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:53.887426 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851b2b14-e25a-47b9-8eab-0e68096e6268-kube-api-access-nh7jv" pod "851b2b14-e25a-47b9-8eab-0e68096e6268" (UID: "851b2b14-e25a-47b9-8eab-0e68096e6268"). InnerVolumeSpecName "kube-api-access-nh7jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:22:53.987217 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:53.987177 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/851b2b14-e25a-47b9-8eab-0e68096e6268-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:53.987217 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:53.987210 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh7jv\" (UniqueName: \"kubernetes.io/projected/851b2b14-e25a-47b9-8eab-0e68096e6268-kube-api-access-nh7jv\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:53.987420 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:53.987225 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/851b2b14-e25a-47b9-8eab-0e68096e6268-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:22:54.629816 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:54.629782 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" Nov 02 18:22:54.630269 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:54.629813 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch" event={"ID":"851b2b14-e25a-47b9-8eab-0e68096e6268","Type":"ContainerDied","Data":"0d176df2dd262c6d63ad22962fe79cc714b0aac81a676d52fd394f8e8c6a8612"} Nov 02 18:22:54.630269 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:54.629880 5601 scope.go:122] "RemoveContainer" containerID="79a03d67e4d2a8e53d948bf06497af0aebcb690ad72d041bfbc0ac55b26a9ce8" Nov 02 18:22:54.654905 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:54.654870 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch"] Nov 02 18:22:54.661214 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:54.661184 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22dfc1-66dc45c696-7p5ch"] Nov 02 18:22:55.149580 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:55.149523 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" podUID="64765539-809c-42a0-af1c-7d1e6b506b18" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:22:55.427907 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:22:55.427872 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="851b2b14-e25a-47b9-8eab-0e68096e6268" path="/var/lib/kubelet/pods/851b2b14-e25a-47b9-8eab-0e68096e6268/volumes" Nov 02 18:23:02.310376 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:02.310335 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b"] Nov 02 18:23:02.310945 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:02.310575 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" podUID="208dcda2-8e4e-4f47-af69-942e7412eda9" containerName="istio-proxy" containerID="cri-o://ae76333d5fbb976dc13fb348e995009d0009545eceabcbf5d19515198fcee852" gracePeriod=30 Nov 02 18:23:03.536429 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:03.536401 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" Nov 02 18:23:03.631685 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:03.631612 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64765539-809c-42a0-af1c-7d1e6b506b18-kube-api-access-gsvzt\" (UniqueName: \"kubernetes.io/projected/64765539-809c-42a0-af1c-7d1e6b506b18-kube-api-access-gsvzt\") pod \"64765539-809c-42a0-af1c-7d1e6b506b18\" (UID: \"64765539-809c-42a0-af1c-7d1e6b506b18\") " Nov 02 18:23:03.631986 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:03.631967 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64765539-809c-42a0-af1c-7d1e6b506b18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64765539-809c-42a0-af1c-7d1e6b506b18-plugins-config-volume\") pod \"64765539-809c-42a0-af1c-7d1e6b506b18\" (UID: \"64765539-809c-42a0-af1c-7d1e6b506b18\") " Nov 02 18:23:03.632376 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:03.632350 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64765539-809c-42a0-af1c-7d1e6b506b18-plugins-config-volume" pod "64765539-809c-42a0-af1c-7d1e6b506b18" (UID: "64765539-809c-42a0-af1c-7d1e6b506b18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:23:03.632498 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:03.632469 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64765539-809c-42a0-af1c-7d1e6b506b18-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:03.633877 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:03.633851 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64765539-809c-42a0-af1c-7d1e6b506b18-kube-api-access-gsvzt" pod "64765539-809c-42a0-af1c-7d1e6b506b18" (UID: "64765539-809c-42a0-af1c-7d1e6b506b18"). InnerVolumeSpecName "kube-api-access-gsvzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:03.734928 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:03.734888 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsvzt\" (UniqueName: \"kubernetes.io/projected/64765539-809c-42a0-af1c-7d1e6b506b18-kube-api-access-gsvzt\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:03.782179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:03.782088 5601 generic.go:356] "Generic (PLEG): container finished" podID="64765539-809c-42a0-af1c-7d1e6b506b18" containerID="1090fa61dc0cbe07ede96e593d5cdeb79c8b0dbc9b256f55a24fbf0840657ef5" exitCode=0 Nov 02 18:23:03.782179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:03.782126 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" event={"ID":"64765539-809c-42a0-af1c-7d1e6b506b18","Type":"ContainerDied","Data":"1090fa61dc0cbe07ede96e593d5cdeb79c8b0dbc9b256f55a24fbf0840657ef5"} Nov 02 18:23:03.782179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:03.782137 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" Nov 02 18:23:03.782179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:03.782177 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5" event={"ID":"64765539-809c-42a0-af1c-7d1e6b506b18","Type":"ContainerDied","Data":"efcc3a15c9244c4cdb23c87363740a3b328cabf56386e6ba1f59a2b460857cd1"} Nov 02 18:23:03.782544 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:03.782203 5601 scope.go:122] "RemoveContainer" containerID="1090fa61dc0cbe07ede96e593d5cdeb79c8b0dbc9b256f55a24fbf0840657ef5" Nov 02 18:23:03.794179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:03.794158 5601 scope.go:122] "RemoveContainer" containerID="1090fa61dc0cbe07ede96e593d5cdeb79c8b0dbc9b256f55a24fbf0840657ef5" Nov 02 18:23:03.794493 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:23:03.794469 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1090fa61dc0cbe07ede96e593d5cdeb79c8b0dbc9b256f55a24fbf0840657ef5\": container with ID starting with 1090fa61dc0cbe07ede96e593d5cdeb79c8b0dbc9b256f55a24fbf0840657ef5 not found: ID does not exist" containerID="1090fa61dc0cbe07ede96e593d5cdeb79c8b0dbc9b256f55a24fbf0840657ef5" Nov 02 18:23:03.794560 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:03.794500 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1090fa61dc0cbe07ede96e593d5cdeb79c8b0dbc9b256f55a24fbf0840657ef5"} err="failed to get container status \"1090fa61dc0cbe07ede96e593d5cdeb79c8b0dbc9b256f55a24fbf0840657ef5\": rpc error: code = NotFound desc = could not find container \"1090fa61dc0cbe07ede96e593d5cdeb79c8b0dbc9b256f55a24fbf0840657ef5\": container with ID starting with 1090fa61dc0cbe07ede96e593d5cdeb79c8b0dbc9b256f55a24fbf0840657ef5 not found: ID does not exist" Nov 02 18:23:03.808994 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:03.808965 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5"] Nov 02 18:23:03.833700 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:03.833605 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22dfc1-64f75878b7-6bhw5"] Nov 02 18:23:04.523144 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:23:04.523105 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:23:05.428267 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:05.428227 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64765539-809c-42a0-af1c-7d1e6b506b18" path="/var/lib/kubelet/pods/64765539-809c-42a0-af1c-7d1e6b506b18/volumes" Nov 02 18:23:07.594405 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.594347 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:23:07.621768 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.621736 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/208dcda2-8e4e-4f47-af69-942e7412eda9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208dcda2-8e4e-4f47-af69-942e7412eda9-istiod-ca-cert\") pod \"208dcda2-8e4e-4f47-af69-942e7412eda9\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " Nov 02 18:23:07.621959 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.621819 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-workload-certs\") pod \"208dcda2-8e4e-4f47-af69-942e7412eda9\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " Nov 02 18:23:07.621959 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.621875 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-podinfo\") pod \"208dcda2-8e4e-4f47-af69-942e7412eda9\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " Nov 02 18:23:07.622092 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.621985 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-credential-socket\") pod \"208dcda2-8e4e-4f47-af69-942e7412eda9\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " Nov 02 18:23:07.622092 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.622076 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-envoy\") pod \"208dcda2-8e4e-4f47-af69-942e7412eda9\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " Nov 02 18:23:07.622208 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.622106 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-data\") pod \"208dcda2-8e4e-4f47-af69-942e7412eda9\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " Nov 02 18:23:07.622208 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.622134 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-workload-socket\") pod \"208dcda2-8e4e-4f47-af69-942e7412eda9\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " Nov 02 18:23:07.622208 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.622144 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/208dcda2-8e4e-4f47-af69-942e7412eda9-istiod-ca-cert" pod "208dcda2-8e4e-4f47-af69-942e7412eda9" (UID: "208dcda2-8e4e-4f47-af69-942e7412eda9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:23:07.622339 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.622243 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-token\" (UniqueName: \"kubernetes.io/projected/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-token\") pod \"208dcda2-8e4e-4f47-af69-942e7412eda9\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " Nov 02 18:23:07.622339 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.622296 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208dcda2-8e4e-4f47-af69-942e7412eda9-kube-api-access-gnkrh\" (UniqueName: \"kubernetes.io/projected/208dcda2-8e4e-4f47-af69-942e7412eda9-kube-api-access-gnkrh\") pod \"208dcda2-8e4e-4f47-af69-942e7412eda9\" (UID: \"208dcda2-8e4e-4f47-af69-942e7412eda9\") " Nov 02 18:23:07.622962 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.622937 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-credential-socket" pod "208dcda2-8e4e-4f47-af69-942e7412eda9" (UID: "208dcda2-8e4e-4f47-af69-942e7412eda9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:07.625158 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.623024 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208dcda2-8e4e-4f47-af69-942e7412eda9-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:07.625158 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.623124 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-data" pod "208dcda2-8e4e-4f47-af69-942e7412eda9" (UID: "208dcda2-8e4e-4f47-af69-942e7412eda9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:07.625158 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.623254 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-workload-socket" pod "208dcda2-8e4e-4f47-af69-942e7412eda9" (UID: "208dcda2-8e4e-4f47-af69-942e7412eda9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:07.625158 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.623525 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-workload-certs" pod "208dcda2-8e4e-4f47-af69-942e7412eda9" (UID: "208dcda2-8e4e-4f47-af69-942e7412eda9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:07.625603 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.625574 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-podinfo" pod "208dcda2-8e4e-4f47-af69-942e7412eda9" (UID: "208dcda2-8e4e-4f47-af69-942e7412eda9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:23:07.625716 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.625647 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-envoy" pod "208dcda2-8e4e-4f47-af69-942e7412eda9" (UID: "208dcda2-8e4e-4f47-af69-942e7412eda9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:07.625780 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.625716 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-token" pod "208dcda2-8e4e-4f47-af69-942e7412eda9" (UID: "208dcda2-8e4e-4f47-af69-942e7412eda9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:07.628093 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.628070 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208dcda2-8e4e-4f47-af69-942e7412eda9-kube-api-access-gnkrh" pod "208dcda2-8e4e-4f47-af69-942e7412eda9" (UID: "208dcda2-8e4e-4f47-af69-942e7412eda9"). InnerVolumeSpecName "kube-api-access-gnkrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:07.725136 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.725101 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:07.725136 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.725134 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:07.725136 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.725143 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:07.725399 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.725153 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:07.725399 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.725164 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnkrh\" (UniqueName: \"kubernetes.io/projected/208dcda2-8e4e-4f47-af69-942e7412eda9-kube-api-access-gnkrh\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:07.725399 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.725172 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:07.725399 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.725181 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208dcda2-8e4e-4f47-af69-942e7412eda9-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:07.725399 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.725190 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208dcda2-8e4e-4f47-af69-942e7412eda9-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:07.813526 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.813487 5601 generic.go:356] "Generic (PLEG): container finished" podID="208dcda2-8e4e-4f47-af69-942e7412eda9" containerID="ae76333d5fbb976dc13fb348e995009d0009545eceabcbf5d19515198fcee852" exitCode=0 Nov 02 18:23:07.813727 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.813543 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" Nov 02 18:23:07.813727 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.813571 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" event={"ID":"208dcda2-8e4e-4f47-af69-942e7412eda9","Type":"ContainerDied","Data":"ae76333d5fbb976dc13fb348e995009d0009545eceabcbf5d19515198fcee852"} Nov 02 18:23:07.813727 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.813680 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b" event={"ID":"208dcda2-8e4e-4f47-af69-942e7412eda9","Type":"ContainerDied","Data":"25d2ca49e62571a2c1c3a622cb9bf48edfd2f16474b7d374ae745d71f176408d"} Nov 02 18:23:07.813727 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.813713 5601 scope.go:122] "RemoveContainer" containerID="ae76333d5fbb976dc13fb348e995009d0009545eceabcbf5d19515198fcee852" Nov 02 18:23:07.826081 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.826042 5601 scope.go:122] "RemoveContainer" containerID="b946e58b3b2011581fc25302f71f1f739853e738a3c262b9eb7833e3307b6d19" Nov 02 18:23:07.842594 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.842569 5601 scope.go:122] "RemoveContainer" containerID="ae76333d5fbb976dc13fb348e995009d0009545eceabcbf5d19515198fcee852" Nov 02 18:23:07.842794 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.842772 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b"] Nov 02 18:23:07.842968 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:23:07.842943 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae76333d5fbb976dc13fb348e995009d0009545eceabcbf5d19515198fcee852\": container with ID starting with ae76333d5fbb976dc13fb348e995009d0009545eceabcbf5d19515198fcee852 not found: ID does not exist" containerID="ae76333d5fbb976dc13fb348e995009d0009545eceabcbf5d19515198fcee852" Nov 02 18:23:07.843020 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.842982 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae76333d5fbb976dc13fb348e995009d0009545eceabcbf5d19515198fcee852"} err="failed to get container status \"ae76333d5fbb976dc13fb348e995009d0009545eceabcbf5d19515198fcee852\": rpc error: code = NotFound desc = could not find container \"ae76333d5fbb976dc13fb348e995009d0009545eceabcbf5d19515198fcee852\": container with ID starting with ae76333d5fbb976dc13fb348e995009d0009545eceabcbf5d19515198fcee852 not found: ID does not exist" Nov 02 18:23:07.843020 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.843006 5601 scope.go:122] "RemoveContainer" containerID="b946e58b3b2011581fc25302f71f1f739853e738a3c262b9eb7833e3307b6d19" Nov 02 18:23:07.843300 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:23:07.843278 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b946e58b3b2011581fc25302f71f1f739853e738a3c262b9eb7833e3307b6d19\": container with ID starting with b946e58b3b2011581fc25302f71f1f739853e738a3c262b9eb7833e3307b6d19 not found: ID does not exist" containerID="b946e58b3b2011581fc25302f71f1f739853e738a3c262b9eb7833e3307b6d19" Nov 02 18:23:07.843355 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.843307 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b946e58b3b2011581fc25302f71f1f739853e738a3c262b9eb7833e3307b6d19"} err="failed to get container status \"b946e58b3b2011581fc25302f71f1f739853e738a3c262b9eb7833e3307b6d19\": rpc error: code = NotFound desc = could not find container \"b946e58b3b2011581fc25302f71f1f739853e738a3c262b9eb7833e3307b6d19\": container with ID starting with b946e58b3b2011581fc25302f71f1f739853e738a3c262b9eb7833e3307b6d19 not found: ID does not exist" Nov 02 18:23:07.847003 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:07.846954 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-22dfc1-openshift-default-79f4d66d95-xw79b"] Nov 02 18:23:09.428297 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:09.428264 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="208dcda2-8e4e-4f47-af69-942e7412eda9" path="/var/lib/kubelet/pods/208dcda2-8e4e-4f47-af69-942e7412eda9/volumes" Nov 02 18:23:11.895581 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:11.895542 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw"] Nov 02 18:23:11.896087 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:11.895752 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" podUID="1bfca870-149d-4258-b915-a50e119a5539" containerName="istio-proxy" containerID="cri-o://18f55657f124542e9f8b7f48f8d6cff3948bf87e8c68601899ae65e704df337e" gracePeriod=30 Nov 02 18:23:17.239290 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.239258 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:23:17.319637 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.319574 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bfca870-149d-4258-b915-a50e119a5539-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfca870-149d-4258-b915-a50e119a5539-istiod-ca-cert\") pod \"1bfca870-149d-4258-b915-a50e119a5539\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " Nov 02 18:23:17.319853 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.319718 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bfca870-149d-4258-b915-a50e119a5539-istio-token\" (UniqueName: \"kubernetes.io/projected/1bfca870-149d-4258-b915-a50e119a5539-istio-token\") pod \"1bfca870-149d-4258-b915-a50e119a5539\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " Nov 02 18:23:17.319853 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.319748 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bfca870-149d-4258-b915-a50e119a5539-kube-api-access-dxf6j\" (UniqueName: \"kubernetes.io/projected/1bfca870-149d-4258-b915-a50e119a5539-kube-api-access-dxf6j\") pod \"1bfca870-149d-4258-b915-a50e119a5539\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " Nov 02 18:23:17.319853 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.319787 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-workload-certs\") pod \"1bfca870-149d-4258-b915-a50e119a5539\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " Nov 02 18:23:17.320347 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.319928 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-istio-data\") pod \"1bfca870-149d-4258-b915-a50e119a5539\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " Nov 02 18:23:17.320347 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.320060 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bfca870-149d-4258-b915-a50e119a5539-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfca870-149d-4258-b915-a50e119a5539-istio-podinfo\") pod \"1bfca870-149d-4258-b915-a50e119a5539\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " Nov 02 18:23:17.320347 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.320124 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-istio-envoy\") pod \"1bfca870-149d-4258-b915-a50e119a5539\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " Nov 02 18:23:17.320347 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.320281 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-credential-socket\") pod \"1bfca870-149d-4258-b915-a50e119a5539\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " Nov 02 18:23:17.321546 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.320386 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-workload-socket\") pod \"1bfca870-149d-4258-b915-a50e119a5539\" (UID: \"1bfca870-149d-4258-b915-a50e119a5539\") " Nov 02 18:23:17.321546 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.321182 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-workload-certs" pod "1bfca870-149d-4258-b915-a50e119a5539" (UID: "1bfca870-149d-4258-b915-a50e119a5539"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:17.321546 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.321276 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-workload-socket" pod "1bfca870-149d-4258-b915-a50e119a5539" (UID: "1bfca870-149d-4258-b915-a50e119a5539"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:17.321546 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.321395 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-credential-socket" pod "1bfca870-149d-4258-b915-a50e119a5539" (UID: "1bfca870-149d-4258-b915-a50e119a5539"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:17.322044 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.322007 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-istio-data" pod "1bfca870-149d-4258-b915-a50e119a5539" (UID: "1bfca870-149d-4258-b915-a50e119a5539"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:17.322335 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.322287 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bfca870-149d-4258-b915-a50e119a5539-istiod-ca-cert" pod "1bfca870-149d-4258-b915-a50e119a5539" (UID: "1bfca870-149d-4258-b915-a50e119a5539"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:23:17.323426 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.323392 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-istio-envoy" pod "1bfca870-149d-4258-b915-a50e119a5539" (UID: "1bfca870-149d-4258-b915-a50e119a5539"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:17.324139 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.324112 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfca870-149d-4258-b915-a50e119a5539-kube-api-access-dxf6j" pod "1bfca870-149d-4258-b915-a50e119a5539" (UID: "1bfca870-149d-4258-b915-a50e119a5539"). InnerVolumeSpecName "kube-api-access-dxf6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:17.325567 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.325540 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfca870-149d-4258-b915-a50e119a5539-istio-token" pod "1bfca870-149d-4258-b915-a50e119a5539" (UID: "1bfca870-149d-4258-b915-a50e119a5539"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:17.326371 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.326322 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bfca870-149d-4258-b915-a50e119a5539-istio-podinfo" pod "1bfca870-149d-4258-b915-a50e119a5539" (UID: "1bfca870-149d-4258-b915-a50e119a5539"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:23:17.422555 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.422516 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:17.422555 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.422557 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfca870-149d-4258-b915-a50e119a5539-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:17.422838 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.422571 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfca870-149d-4258-b915-a50e119a5539-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:17.422838 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.422586 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxf6j\" (UniqueName: \"kubernetes.io/projected/1bfca870-149d-4258-b915-a50e119a5539-kube-api-access-dxf6j\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:17.422838 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.422603 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:17.422838 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.422630 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:17.422838 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.422645 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfca870-149d-4258-b915-a50e119a5539-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:17.422838 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.422660 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:17.422838 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.422673 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfca870-149d-4258-b915-a50e119a5539-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:17.930632 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.930587 5601 generic.go:356] "Generic (PLEG): container finished" podID="1bfca870-149d-4258-b915-a50e119a5539" containerID="18f55657f124542e9f8b7f48f8d6cff3948bf87e8c68601899ae65e704df337e" exitCode=0 Nov 02 18:23:17.930846 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.930670 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" Nov 02 18:23:17.930846 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.930687 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" event={"ID":"1bfca870-149d-4258-b915-a50e119a5539","Type":"ContainerDied","Data":"18f55657f124542e9f8b7f48f8d6cff3948bf87e8c68601899ae65e704df337e"} Nov 02 18:23:17.930846 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.930761 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw" event={"ID":"1bfca870-149d-4258-b915-a50e119a5539","Type":"ContainerDied","Data":"3929788826e02f2369d71eeb2f255eab983c8447dd5e9e0ae47cb2507f05ec15"} Nov 02 18:23:17.930846 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.930791 5601 scope.go:122] "RemoveContainer" containerID="18f55657f124542e9f8b7f48f8d6cff3948bf87e8c68601899ae65e704df337e" Nov 02 18:23:17.943676 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.943645 5601 scope.go:122] "RemoveContainer" containerID="18f55657f124542e9f8b7f48f8d6cff3948bf87e8c68601899ae65e704df337e" Nov 02 18:23:17.944017 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:23:17.943993 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f55657f124542e9f8b7f48f8d6cff3948bf87e8c68601899ae65e704df337e\": container with ID starting with 18f55657f124542e9f8b7f48f8d6cff3948bf87e8c68601899ae65e704df337e not found: ID does not exist" containerID="18f55657f124542e9f8b7f48f8d6cff3948bf87e8c68601899ae65e704df337e" Nov 02 18:23:17.944121 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.944026 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f55657f124542e9f8b7f48f8d6cff3948bf87e8c68601899ae65e704df337e"} err="failed to get container status \"18f55657f124542e9f8b7f48f8d6cff3948bf87e8c68601899ae65e704df337e\": rpc error: code = NotFound desc = could not find container \"18f55657f124542e9f8b7f48f8d6cff3948bf87e8c68601899ae65e704df337e\": container with ID starting with 18f55657f124542e9f8b7f48f8d6cff3948bf87e8c68601899ae65e704df337e not found: ID does not exist" Nov 02 18:23:17.956406 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.956374 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw"] Nov 02 18:23:17.964311 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:17.964281 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-fx2zw"] Nov 02 18:23:19.427839 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:19.427803 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bfca870-149d-4258-b915-a50e119a5539" path="/var/lib/kubelet/pods/1bfca870-149d-4258-b915-a50e119a5539/volumes" Nov 02 18:23:22.129204 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:22.129165 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf"] Nov 02 18:23:22.129833 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:22.129438 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" podUID="5be85144-c6c7-4399-9837-737b76d4e114" containerName="maas-api" containerID="cri-o://6e5d55db4f5ff09a020f2777d2d21c044f5cd75e1e68cab8e587b60800246552" gracePeriod=30 Nov 02 18:23:22.740657 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:22.740628 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" Nov 02 18:23:22.883550 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:22.883516 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be85144-c6c7-4399-9837-737b76d4e114-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be85144-c6c7-4399-9837-737b76d4e114-openshift-service-ca\") pod \"5be85144-c6c7-4399-9837-737b76d4e114\" (UID: \"5be85144-c6c7-4399-9837-737b76d4e114\") " Nov 02 18:23:22.883550 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:22.883553 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be85144-c6c7-4399-9837-737b76d4e114-kube-api-access-kbmz8\" (UniqueName: \"kubernetes.io/projected/5be85144-c6c7-4399-9837-737b76d4e114-kube-api-access-kbmz8\") pod \"5be85144-c6c7-4399-9837-737b76d4e114\" (UID: \"5be85144-c6c7-4399-9837-737b76d4e114\") " Nov 02 18:23:22.883829 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:22.883664 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5be85144-c6c7-4399-9837-737b76d4e114-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5be85144-c6c7-4399-9837-737b76d4e114-metrics-certs\") pod \"5be85144-c6c7-4399-9837-737b76d4e114\" (UID: \"5be85144-c6c7-4399-9837-737b76d4e114\") " Nov 02 18:23:22.883951 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:22.883930 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5be85144-c6c7-4399-9837-737b76d4e114-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5be85144-c6c7-4399-9837-737b76d4e114-maas-api-tls\") pod \"5be85144-c6c7-4399-9837-737b76d4e114\" (UID: \"5be85144-c6c7-4399-9837-737b76d4e114\") " Nov 02 18:23:22.884012 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:22.883975 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be85144-c6c7-4399-9837-737b76d4e114-openshift-service-ca" pod "5be85144-c6c7-4399-9837-737b76d4e114" (UID: "5be85144-c6c7-4399-9837-737b76d4e114"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:23:22.884693 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:22.884666 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be85144-c6c7-4399-9837-737b76d4e114-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:22.885967 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:22.885933 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5be85144-c6c7-4399-9837-737b76d4e114-metrics-certs" pod "5be85144-c6c7-4399-9837-737b76d4e114" (UID: "5be85144-c6c7-4399-9837-737b76d4e114"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:23:22.885967 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:22.885933 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be85144-c6c7-4399-9837-737b76d4e114-kube-api-access-kbmz8" pod "5be85144-c6c7-4399-9837-737b76d4e114" (UID: "5be85144-c6c7-4399-9837-737b76d4e114"). InnerVolumeSpecName "kube-api-access-kbmz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:22.886408 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:22.886382 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5be85144-c6c7-4399-9837-737b76d4e114-maas-api-tls" pod "5be85144-c6c7-4399-9837-737b76d4e114" (UID: "5be85144-c6c7-4399-9837-737b76d4e114"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:23:22.986236 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:22.986150 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbmz8\" (UniqueName: \"kubernetes.io/projected/5be85144-c6c7-4399-9837-737b76d4e114-kube-api-access-kbmz8\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:22.986236 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:22.986182 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5be85144-c6c7-4399-9837-737b76d4e114-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:22.986236 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:22.986193 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5be85144-c6c7-4399-9837-737b76d4e114-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:23.032643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.014547 5601 generic.go:356] "Generic (PLEG): container finished" podID="5be85144-c6c7-4399-9837-737b76d4e114" containerID="6e5d55db4f5ff09a020f2777d2d21c044f5cd75e1e68cab8e587b60800246552" exitCode=0 Nov 02 18:23:23.032643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.014705 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" event={"ID":"5be85144-c6c7-4399-9837-737b76d4e114","Type":"ContainerDied","Data":"6e5d55db4f5ff09a020f2777d2d21c044f5cd75e1e68cab8e587b60800246552"} Nov 02 18:23:23.032643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.014762 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" event={"ID":"5be85144-c6c7-4399-9837-737b76d4e114","Type":"ContainerDied","Data":"78806d11660f9dcf0378ab5a7b3818e3c6913e41beec1318e1c5d17e6d2a8799"} Nov 02 18:23:23.032643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.014795 5601 scope.go:122] "RemoveContainer" containerID="6e5d55db4f5ff09a020f2777d2d21c044f5cd75e1e68cab8e587b60800246552" Nov 02 18:23:23.032643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.014941 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf" Nov 02 18:23:23.180890 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.180706 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf"] Nov 02 18:23:23.188746 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.186765 5601 scope.go:122] "RemoveContainer" containerID="6e5d55db4f5ff09a020f2777d2d21c044f5cd75e1e68cab8e587b60800246552" Nov 02 18:23:23.191677 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:23:23.190789 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5d55db4f5ff09a020f2777d2d21c044f5cd75e1e68cab8e587b60800246552\": container with ID starting with 6e5d55db4f5ff09a020f2777d2d21c044f5cd75e1e68cab8e587b60800246552 not found: ID does not exist" containerID="6e5d55db4f5ff09a020f2777d2d21c044f5cd75e1e68cab8e587b60800246552" Nov 02 18:23:23.191677 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.190838 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5d55db4f5ff09a020f2777d2d21c044f5cd75e1e68cab8e587b60800246552"} err="failed to get container status \"6e5d55db4f5ff09a020f2777d2d21c044f5cd75e1e68cab8e587b60800246552\": rpc error: code = NotFound desc = could not find container \"6e5d55db4f5ff09a020f2777d2d21c044f5cd75e1e68cab8e587b60800246552\": container with ID starting with 6e5d55db4f5ff09a020f2777d2d21c044f5cd75e1e68cab8e587b60800246552 not found: ID does not exist" Nov 02 18:23:23.196305 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.196275 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-63be7a-79b594b699-px7cf"] Nov 02 18:23:23.428333 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.428300 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5be85144-c6c7-4399-9837-737b76d4e114" path="/var/lib/kubelet/pods/5be85144-c6c7-4399-9837-737b76d4e114/volumes" Nov 02 18:23:23.492604 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.492570 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8"] Nov 02 18:23:23.493555 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493532 5601 cpu_manager.go:414] "containerMap: removing container" podUID="a15cd03a-9403-4912-a060-4ae6dab8baf8" containerName="istio-proxy" Nov 02 18:23:23.493555 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493555 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15cd03a-9403-4912-a060-4ae6dab8baf8" containerName="istio-proxy" Nov 02 18:23:23.493850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493582 5601 cpu_manager.go:414] "containerMap: removing container" podUID="1bfca870-149d-4258-b915-a50e119a5539" containerName="istio-proxy" Nov 02 18:23:23.493850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493591 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bfca870-149d-4258-b915-a50e119a5539" containerName="istio-proxy" Nov 02 18:23:23.493850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493609 5601 cpu_manager.go:414] "containerMap: removing container" podUID="945a032d-c69f-4d9e-96a5-db1ac483af67" containerName="istio-proxy" Nov 02 18:23:23.493850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493629 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945a032d-c69f-4d9e-96a5-db1ac483af67" containerName="istio-proxy" Nov 02 18:23:23.493850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493641 5601 cpu_manager.go:414] "containerMap: removing container" podUID="945a032d-c69f-4d9e-96a5-db1ac483af67" containerName="istio-proxy" Nov 02 18:23:23.493850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493648 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945a032d-c69f-4d9e-96a5-db1ac483af67" containerName="istio-proxy" Nov 02 18:23:23.493850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493663 5601 cpu_manager.go:414] "containerMap: removing container" podUID="5be85144-c6c7-4399-9837-737b76d4e114" containerName="maas-api" Nov 02 18:23:23.493850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493670 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be85144-c6c7-4399-9837-737b76d4e114" containerName="maas-api" Nov 02 18:23:23.493850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493692 5601 cpu_manager.go:414] "containerMap: removing container" podUID="ab48f110-7cc1-45fb-a416-527d9c919917" containerName="main" Nov 02 18:23:23.493850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493699 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab48f110-7cc1-45fb-a416-527d9c919917" containerName="main" Nov 02 18:23:23.493850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493711 5601 cpu_manager.go:414] "containerMap: removing container" podUID="851b2b14-e25a-47b9-8eab-0e68096e6268" containerName="maas-api" Nov 02 18:23:23.493850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493719 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="851b2b14-e25a-47b9-8eab-0e68096e6268" containerName="maas-api" Nov 02 18:23:23.493850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493734 5601 cpu_manager.go:414] "containerMap: removing container" podUID="1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5" containerName="payload-pre-processing" Nov 02 18:23:23.493850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493741 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5" containerName="payload-pre-processing" Nov 02 18:23:23.493850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493758 5601 cpu_manager.go:414] "containerMap: removing container" podUID="5717d304-fd02-42a4-a04f-0958ce43fdb0" containerName="maas-api" Nov 02 18:23:23.493850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493765 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5717d304-fd02-42a4-a04f-0958ce43fdb0" containerName="maas-api" Nov 02 18:23:23.493850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493783 5601 cpu_manager.go:414] "containerMap: removing container" podUID="5c2aa306-9a80-4299-9ada-d7229e813dee" containerName="istio-proxy" Nov 02 18:23:23.493850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493832 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2aa306-9a80-4299-9ada-d7229e813dee" containerName="istio-proxy" Nov 02 18:23:23.493850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493856 5601 cpu_manager.go:414] "containerMap: removing container" podUID="64765539-809c-42a0-af1c-7d1e6b506b18" containerName="payload-pre-processing" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493864 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64765539-809c-42a0-af1c-7d1e6b506b18" containerName="payload-pre-processing" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493877 5601 cpu_manager.go:414] "containerMap: removing container" podUID="ae604043-587b-4fd4-b923-4bd1b6170df2" containerName="payload-pre-processing" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493884 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae604043-587b-4fd4-b923-4bd1b6170df2" containerName="payload-pre-processing" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493898 5601 cpu_manager.go:414] "containerMap: removing container" podUID="e162853a-15f2-42ad-b350-bbc126c32272" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493906 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e162853a-15f2-42ad-b350-bbc126c32272" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493914 5601 cpu_manager.go:414] "containerMap: removing container" podUID="196bcedf-fdfd-4469-9234-d281f907d53a" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493920 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="196bcedf-fdfd-4469-9234-d281f907d53a" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493933 5601 cpu_manager.go:414] "containerMap: removing container" podUID="5c2aa306-9a80-4299-9ada-d7229e813dee" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493939 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2aa306-9a80-4299-9ada-d7229e813dee" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493971 5601 cpu_manager.go:414] "containerMap: removing container" podUID="208dcda2-8e4e-4f47-af69-942e7412eda9" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493979 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208dcda2-8e4e-4f47-af69-942e7412eda9" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.493993 5601 cpu_manager.go:414] "containerMap: removing container" podUID="208dcda2-8e4e-4f47-af69-942e7412eda9" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.494001 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208dcda2-8e4e-4f47-af69-942e7412eda9" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.494182 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bfca870-149d-4258-b915-a50e119a5539" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.494199 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="945a032d-c69f-4d9e-96a5-db1ac483af67" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.494211 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="945a032d-c69f-4d9e-96a5-db1ac483af67" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.494223 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="5be85144-c6c7-4399-9837-737b76d4e114" containerName="maas-api" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.494234 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab48f110-7cc1-45fb-a416-527d9c919917" containerName="main" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.494243 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="851b2b14-e25a-47b9-8eab-0e68096e6268" containerName="maas-api" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.494257 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3c2c03-7ac1-4ca4-9c79-3f0bdf8bc5e5" containerName="payload-pre-processing" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.494267 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="5717d304-fd02-42a4-a04f-0958ce43fdb0" containerName="maas-api" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.494275 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="84ed3fb1-d6b4-469c-b7a3-8af367d47208" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.494287 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2aa306-9a80-4299-9ada-d7229e813dee" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.494296 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="64765539-809c-42a0-af1c-7d1e6b506b18" containerName="payload-pre-processing" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.494305 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae604043-587b-4fd4-b923-4bd1b6170df2" containerName="payload-pre-processing" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.494315 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="e162853a-15f2-42ad-b350-bbc126c32272" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.494324 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="196bcedf-fdfd-4469-9234-d281f907d53a" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.494333 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2aa306-9a80-4299-9ada-d7229e813dee" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.494347 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="208dcda2-8e4e-4f47-af69-942e7412eda9" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.494358 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="208dcda2-8e4e-4f47-af69-942e7412eda9" containerName="istio-proxy" Nov 02 18:23:23.494867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.494372 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15cd03a-9403-4912-a060-4ae6dab8baf8" containerName="istio-proxy" Nov 02 18:23:23.500084 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.500063 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:23.504909 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.504884 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2966e7\"/\"test-model-2966e7-kserve-self-signed-certs\"" Nov 02 18:23:23.507703 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.507488 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2966e7\"/\"default-dockercfg-b4c97\"" Nov 02 18:23:23.510844 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.510491 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8"] Nov 02 18:23:23.515553 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.515527 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-dshm\") pod \"test-model-2966e7-kserve-896499f7c-lfxs8\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:23.515721 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.515570 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-tls-certs\") pod \"test-model-2966e7-kserve-896499f7c-lfxs8\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:23.515816 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.515796 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-model-cache\") pod \"test-model-2966e7-kserve-896499f7c-lfxs8\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:23.516019 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.515900 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-tmp-dir\") pod \"test-model-2966e7-kserve-896499f7c-lfxs8\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:23.516129 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.516046 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl992\" (UniqueName: \"kubernetes.io/projected/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-kube-api-access-dl992\") pod \"test-model-2966e7-kserve-896499f7c-lfxs8\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:23.516129 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.516093 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-home\") pod \"test-model-2966e7-kserve-896499f7c-lfxs8\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:23.517579 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.517542 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2966e7\"/\"kube-root-ca.crt\"" Nov 02 18:23:23.517579 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.517561 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2966e7\"/\"openshift-service-ca.crt\"" Nov 02 18:23:23.618010 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.617959 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-dshm\") pod \"test-model-2966e7-kserve-896499f7c-lfxs8\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:23.618220 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.618031 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-tls-certs\") pod \"test-model-2966e7-kserve-896499f7c-lfxs8\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:23.618220 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.618183 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-model-cache\") pod \"test-model-2966e7-kserve-896499f7c-lfxs8\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:23.618590 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.618564 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-model-cache\") pod \"test-model-2966e7-kserve-896499f7c-lfxs8\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:23.618780 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.618760 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-tmp-dir\") pod \"test-model-2966e7-kserve-896499f7c-lfxs8\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:23.619185 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.619091 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-tmp-dir\") pod \"test-model-2966e7-kserve-896499f7c-lfxs8\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:23.619300 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.619254 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl992\" (UniqueName: \"kubernetes.io/projected/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-kube-api-access-dl992\") pod \"test-model-2966e7-kserve-896499f7c-lfxs8\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:23.620283 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.619635 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-home\") pod \"test-model-2966e7-kserve-896499f7c-lfxs8\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:23.620283 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.620043 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-home\") pod \"test-model-2966e7-kserve-896499f7c-lfxs8\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:23.621039 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.621013 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-dshm\") pod \"test-model-2966e7-kserve-896499f7c-lfxs8\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:23.621338 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.621318 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-tls-certs\") pod \"test-model-2966e7-kserve-896499f7c-lfxs8\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:23.627887 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.627607 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl992\" (UniqueName: \"kubernetes.io/projected/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-kube-api-access-dl992\") pod \"test-model-2966e7-kserve-896499f7c-lfxs8\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:23.824490 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:23.818177 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:24.253391 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:24.253339 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8"] Nov 02 18:23:24.255304 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:23:24.255239 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e88c53c_c96d_4e18_9cf1_2d54e8f55f1f.slice/crio-b605734a9f8826c23132bd598c6d84dc1a1379ae5801955f337cab44f89b400a WatchSource:0}: Error finding container b605734a9f8826c23132bd598c6d84dc1a1379ae5801955f337cab44f89b400a: Status 404 returned error can't find the container with id b605734a9f8826c23132bd598c6d84dc1a1379ae5801955f337cab44f89b400a Nov 02 18:23:24.860107 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:24.860064 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84"] Nov 02 18:23:24.860644 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:24.860566 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" podUID="3b429f46-034e-43a5-bffb-65f58b9c84a4" containerName="istio-proxy" containerID="cri-o://2c6ab9f2831902ede4cb641a12205663b57da652786eb15d6897b6b47884b9ff" gracePeriod=30 Nov 02 18:23:25.115700 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:25.106756 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" event={"ID":"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f","Type":"ContainerStarted","Data":"a87c32e75680034091af1161730fdeaf8af7bf2513e5aee74b912fb56093a52a"} Nov 02 18:23:25.115700 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:25.106816 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" event={"ID":"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f","Type":"ContainerStarted","Data":"b605734a9f8826c23132bd598c6d84dc1a1379ae5801955f337cab44f89b400a"} Nov 02 18:23:25.820199 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:25.820154 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:30.169759 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.169724 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:23:30.189004 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.188961 5601 generic.go:356] "Generic (PLEG): container finished" podID="3b429f46-034e-43a5-bffb-65f58b9c84a4" containerID="2c6ab9f2831902ede4cb641a12205663b57da652786eb15d6897b6b47884b9ff" exitCode=0 Nov 02 18:23:30.189177 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.189014 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" Nov 02 18:23:30.189177 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.189044 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" event={"ID":"3b429f46-034e-43a5-bffb-65f58b9c84a4","Type":"ContainerDied","Data":"2c6ab9f2831902ede4cb641a12205663b57da652786eb15d6897b6b47884b9ff"} Nov 02 18:23:30.189177 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.189102 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84" event={"ID":"3b429f46-034e-43a5-bffb-65f58b9c84a4","Type":"ContainerDied","Data":"342b671ae45e65bd10856367a5c4f0bfe524dbc307fc433e7af1fba802baed13"} Nov 02 18:23:30.189177 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.189128 5601 scope.go:122] "RemoveContainer" containerID="2c6ab9f2831902ede4cb641a12205663b57da652786eb15d6897b6b47884b9ff" Nov 02 18:23:30.195154 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.195095 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" podStartSLOduration=7.195075886 podStartE2EDuration="7.195075886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:23:25.170746924 +0000 UTC m=+2969.861800938" watchObservedRunningTime="2026-11-02 18:23:30.195075886 +0000 UTC m=+2974.886129898" Nov 02 18:23:30.202628 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.202589 5601 scope.go:122] "RemoveContainer" containerID="2c6ab9f2831902ede4cb641a12205663b57da652786eb15d6897b6b47884b9ff" Nov 02 18:23:30.202993 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:23:30.202968 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c6ab9f2831902ede4cb641a12205663b57da652786eb15d6897b6b47884b9ff\": container with ID starting with 2c6ab9f2831902ede4cb641a12205663b57da652786eb15d6897b6b47884b9ff not found: ID does not exist" containerID="2c6ab9f2831902ede4cb641a12205663b57da652786eb15d6897b6b47884b9ff" Nov 02 18:23:30.203069 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.203005 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c6ab9f2831902ede4cb641a12205663b57da652786eb15d6897b6b47884b9ff"} err="failed to get container status \"2c6ab9f2831902ede4cb641a12205663b57da652786eb15d6897b6b47884b9ff\": rpc error: code = NotFound desc = could not find container \"2c6ab9f2831902ede4cb641a12205663b57da652786eb15d6897b6b47884b9ff\": container with ID starting with 2c6ab9f2831902ede4cb641a12205663b57da652786eb15d6897b6b47884b9ff not found: ID does not exist" Nov 02 18:23:30.279656 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.279599 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-workload-certs\") pod \"3b429f46-034e-43a5-bffb-65f58b9c84a4\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " Nov 02 18:23:30.279909 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.279749 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b429f46-034e-43a5-bffb-65f58b9c84a4-kube-api-access-smtf9\" (UniqueName: \"kubernetes.io/projected/3b429f46-034e-43a5-bffb-65f58b9c84a4-kube-api-access-smtf9\") pod \"3b429f46-034e-43a5-bffb-65f58b9c84a4\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " Nov 02 18:23:30.279909 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.279825 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-workload-socket\") pod \"3b429f46-034e-43a5-bffb-65f58b9c84a4\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " Nov 02 18:23:30.279909 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.279892 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-podinfo\") pod \"3b429f46-034e-43a5-bffb-65f58b9c84a4\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " Nov 02 18:23:30.280182 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.279957 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-data\") pod \"3b429f46-034e-43a5-bffb-65f58b9c84a4\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " Nov 02 18:23:30.280182 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.279994 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-token\" (UniqueName: \"kubernetes.io/projected/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-token\") pod \"3b429f46-034e-43a5-bffb-65f58b9c84a4\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " Nov 02 18:23:30.280182 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.280069 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b429f46-034e-43a5-bffb-65f58b9c84a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b429f46-034e-43a5-bffb-65f58b9c84a4-istiod-ca-cert\") pod \"3b429f46-034e-43a5-bffb-65f58b9c84a4\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " Nov 02 18:23:30.280182 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.280102 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-credential-socket\") pod \"3b429f46-034e-43a5-bffb-65f58b9c84a4\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " Nov 02 18:23:30.286503 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.280230 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-envoy\") pod \"3b429f46-034e-43a5-bffb-65f58b9c84a4\" (UID: \"3b429f46-034e-43a5-bffb-65f58b9c84a4\") " Nov 02 18:23:30.286503 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.282220 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-workload-socket" pod "3b429f46-034e-43a5-bffb-65f58b9c84a4" (UID: "3b429f46-034e-43a5-bffb-65f58b9c84a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.286503 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.283258 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-data" pod "3b429f46-034e-43a5-bffb-65f58b9c84a4" (UID: "3b429f46-034e-43a5-bffb-65f58b9c84a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.286503 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.283348 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-credential-socket" pod "3b429f46-034e-43a5-bffb-65f58b9c84a4" (UID: "3b429f46-034e-43a5-bffb-65f58b9c84a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.286503 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.283550 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b429f46-034e-43a5-bffb-65f58b9c84a4-istiod-ca-cert" pod "3b429f46-034e-43a5-bffb-65f58b9c84a4" (UID: "3b429f46-034e-43a5-bffb-65f58b9c84a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:23:30.286503 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.283811 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-workload-certs" pod "3b429f46-034e-43a5-bffb-65f58b9c84a4" (UID: "3b429f46-034e-43a5-bffb-65f58b9c84a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.286503 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.285093 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-envoy" pod "3b429f46-034e-43a5-bffb-65f58b9c84a4" (UID: "3b429f46-034e-43a5-bffb-65f58b9c84a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:30.286503 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.285932 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-podinfo" pod "3b429f46-034e-43a5-bffb-65f58b9c84a4" (UID: "3b429f46-034e-43a5-bffb-65f58b9c84a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:23:30.286503 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.286005 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-token" pod "3b429f46-034e-43a5-bffb-65f58b9c84a4" (UID: "3b429f46-034e-43a5-bffb-65f58b9c84a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:30.286503 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.286197 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b429f46-034e-43a5-bffb-65f58b9c84a4-kube-api-access-smtf9" pod "3b429f46-034e-43a5-bffb-65f58b9c84a4" (UID: "3b429f46-034e-43a5-bffb-65f58b9c84a4"). InnerVolumeSpecName "kube-api-access-smtf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:30.384940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.384786 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b429f46-034e-43a5-bffb-65f58b9c84a4-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.384940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.384821 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.384940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.384836 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.384940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.384850 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.384940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.384866 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smtf9\" (UniqueName: \"kubernetes.io/projected/3b429f46-034e-43a5-bffb-65f58b9c84a4-kube-api-access-smtf9\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.384940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.384880 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.384940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.384894 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.384940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.384907 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.384940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.384923 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b429f46-034e-43a5-bffb-65f58b9c84a4-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:23:30.523852 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.523819 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84"] Nov 02 18:23:30.529444 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:30.529411 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-45b84"] Nov 02 18:23:31.470944 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:31.470903 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b429f46-034e-43a5-bffb-65f58b9c84a4" path="/var/lib/kubelet/pods/3b429f46-034e-43a5-bffb-65f58b9c84a4/volumes" Nov 02 18:23:33.837711 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:33.833026 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:33.915641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:33.900982 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:34.233696 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:23:34.233637 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:23:34.488163 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:23:34.488076 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:24:04.487190 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:24:04.487143 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:24:31.808005 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:24:31.807735 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-jv8f4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:24:31.808005 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:24:31.807788 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" podUID="c32fa6f3-9665-4b3b-b093-22435190829b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:24:34.487311 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:24:34.487269 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:24:55.686079 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:24:55.686042 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl"] Nov 02 18:24:55.686519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:24:55.686239 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" podUID="72a481ce-cb98-41e1-bb69-db80124ba5c4" containerName="istio-proxy" containerID="cri-o://1fe98daf3a74a709e4afdd3b0c298b6666de9854f14863d2091411d57d1598e5" gracePeriod=30 Nov 02 18:25:00.959071 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:00.958989 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 18:25:01.027230 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.027183 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a481ce-cb98-41e1-bb69-db80124ba5c4-kube-api-access-g2fwx\" (UniqueName: \"kubernetes.io/projected/72a481ce-cb98-41e1-bb69-db80124ba5c4-kube-api-access-g2fwx\") pod \"72a481ce-cb98-41e1-bb69-db80124ba5c4\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " Nov 02 18:25:01.027472 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.027246 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-workload-certs\") pod \"72a481ce-cb98-41e1-bb69-db80124ba5c4\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " Nov 02 18:25:01.027472 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.027325 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-token\" (UniqueName: \"kubernetes.io/projected/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-token\") pod \"72a481ce-cb98-41e1-bb69-db80124ba5c4\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " Nov 02 18:25:01.027472 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.027377 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-envoy\") pod \"72a481ce-cb98-41e1-bb69-db80124ba5c4\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " Nov 02 18:25:01.027708 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.027534 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-podinfo\") pod \"72a481ce-cb98-41e1-bb69-db80124ba5c4\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " Nov 02 18:25:01.027708 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.027578 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-workload-certs" pod "72a481ce-cb98-41e1-bb69-db80124ba5c4" (UID: "72a481ce-cb98-41e1-bb69-db80124ba5c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:01.027708 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.027693 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-credential-socket\") pod \"72a481ce-cb98-41e1-bb69-db80124ba5c4\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " Nov 02 18:25:01.027880 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.027723 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-workload-socket\") pod \"72a481ce-cb98-41e1-bb69-db80124ba5c4\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " Nov 02 18:25:01.027880 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.027753 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-data\") pod \"72a481ce-cb98-41e1-bb69-db80124ba5c4\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " Nov 02 18:25:01.027880 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.027819 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a481ce-cb98-41e1-bb69-db80124ba5c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a481ce-cb98-41e1-bb69-db80124ba5c4-istiod-ca-cert\") pod \"72a481ce-cb98-41e1-bb69-db80124ba5c4\" (UID: \"72a481ce-cb98-41e1-bb69-db80124ba5c4\") " Nov 02 18:25:01.028045 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.028027 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-credential-socket" pod "72a481ce-cb98-41e1-bb69-db80124ba5c4" (UID: "72a481ce-cb98-41e1-bb69-db80124ba5c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:01.028379 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.028354 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a481ce-cb98-41e1-bb69-db80124ba5c4-istiod-ca-cert" pod "72a481ce-cb98-41e1-bb69-db80124ba5c4" (UID: "72a481ce-cb98-41e1-bb69-db80124ba5c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:25:01.028483 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.028406 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-data" pod "72a481ce-cb98-41e1-bb69-db80124ba5c4" (UID: "72a481ce-cb98-41e1-bb69-db80124ba5c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:01.028570 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.028550 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:01.028570 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.028569 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:01.028786 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.028583 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a481ce-cb98-41e1-bb69-db80124ba5c4-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:01.028786 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.028595 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:01.028905 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.028833 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-workload-socket" pod "72a481ce-cb98-41e1-bb69-db80124ba5c4" (UID: "72a481ce-cb98-41e1-bb69-db80124ba5c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:01.030426 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.030400 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a481ce-cb98-41e1-bb69-db80124ba5c4-kube-api-access-g2fwx" pod "72a481ce-cb98-41e1-bb69-db80124ba5c4" (UID: "72a481ce-cb98-41e1-bb69-db80124ba5c4"). InnerVolumeSpecName "kube-api-access-g2fwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:25:01.030553 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.030485 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-token" pod "72a481ce-cb98-41e1-bb69-db80124ba5c4" (UID: "72a481ce-cb98-41e1-bb69-db80124ba5c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:25:01.030991 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.030969 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-podinfo" pod "72a481ce-cb98-41e1-bb69-db80124ba5c4" (UID: "72a481ce-cb98-41e1-bb69-db80124ba5c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:25:01.031074 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.031020 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-envoy" pod "72a481ce-cb98-41e1-bb69-db80124ba5c4" (UID: "72a481ce-cb98-41e1-bb69-db80124ba5c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:01.095059 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.094488 5601 generic.go:356] "Generic (PLEG): container finished" podID="72a481ce-cb98-41e1-bb69-db80124ba5c4" containerID="1fe98daf3a74a709e4afdd3b0c298b6666de9854f14863d2091411d57d1598e5" exitCode=0 Nov 02 18:25:01.095059 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.094769 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" event={"ID":"72a481ce-cb98-41e1-bb69-db80124ba5c4","Type":"ContainerDied","Data":"1fe98daf3a74a709e4afdd3b0c298b6666de9854f14863d2091411d57d1598e5"} Nov 02 18:25:01.095059 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.094833 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" event={"ID":"72a481ce-cb98-41e1-bb69-db80124ba5c4","Type":"ContainerDied","Data":"eed6ce29ecb8c06f1484efb0ea5ce3f7cccc47b5cd9a2c26a22b508c71f8c935"} Nov 02 18:25:01.095059 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.094866 5601 scope.go:122] "RemoveContainer" containerID="1fe98daf3a74a709e4afdd3b0c298b6666de9854f14863d2091411d57d1598e5" Nov 02 18:25:01.100145 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.098977 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl" Nov 02 18:25:01.118612 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.118585 5601 scope.go:122] "RemoveContainer" containerID="1fe98daf3a74a709e4afdd3b0c298b6666de9854f14863d2091411d57d1598e5" Nov 02 18:25:01.118955 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:25:01.118929 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe98daf3a74a709e4afdd3b0c298b6666de9854f14863d2091411d57d1598e5\": container with ID starting with 1fe98daf3a74a709e4afdd3b0c298b6666de9854f14863d2091411d57d1598e5 not found: ID does not exist" containerID="1fe98daf3a74a709e4afdd3b0c298b6666de9854f14863d2091411d57d1598e5" Nov 02 18:25:01.119039 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.118967 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe98daf3a74a709e4afdd3b0c298b6666de9854f14863d2091411d57d1598e5"} err="failed to get container status \"1fe98daf3a74a709e4afdd3b0c298b6666de9854f14863d2091411d57d1598e5\": rpc error: code = NotFound desc = could not find container \"1fe98daf3a74a709e4afdd3b0c298b6666de9854f14863d2091411d57d1598e5\": container with ID starting with 1fe98daf3a74a709e4afdd3b0c298b6666de9854f14863d2091411d57d1598e5 not found: ID does not exist" Nov 02 18:25:01.129496 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.129468 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl"] Nov 02 18:25:01.130404 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.130383 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:01.130463 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.130407 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:01.130463 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.130420 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a481ce-cb98-41e1-bb69-db80124ba5c4-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:01.130463 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.130429 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a481ce-cb98-41e1-bb69-db80124ba5c4-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:01.130463 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.130438 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2fwx\" (UniqueName: \"kubernetes.io/projected/72a481ce-cb98-41e1-bb69-db80124ba5c4-kube-api-access-g2fwx\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:01.142814 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.142778 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-rcvxl"] Nov 02 18:25:01.435624 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:01.435135 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a481ce-cb98-41e1-bb69-db80124ba5c4" path="/var/lib/kubelet/pods/72a481ce-cb98-41e1-bb69-db80124ba5c4/volumes" Nov 02 18:25:04.486731 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:25:04.486689 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:25:06.524350 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:06.524305 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7"] Nov 02 18:25:21.552407 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:21.552334 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" podUID="dd384d95-642a-4f20-9981-4dc6eb903c7f" containerName="main" containerID="cri-o://5f2492066dbf75ce2e382c3cf5f9fb7549b489831a6bd8b67bf49e17b93adb12" gracePeriod=45 Nov 02 18:25:21.847485 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:21.847455 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:25:21.899375 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:21.899344 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd384d95-642a-4f20-9981-4dc6eb903c7f-tls-certs\" (UniqueName: \"kubernetes.io/secret/dd384d95-642a-4f20-9981-4dc6eb903c7f-tls-certs\") pod \"dd384d95-642a-4f20-9981-4dc6eb903c7f\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " Nov 02 18:25:21.899556 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:21.899467 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-dshm\") pod \"dd384d95-642a-4f20-9981-4dc6eb903c7f\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " Nov 02 18:25:21.899556 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:21.899516 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd384d95-642a-4f20-9981-4dc6eb903c7f-kube-api-access-zl62d\" (UniqueName: \"kubernetes.io/projected/dd384d95-642a-4f20-9981-4dc6eb903c7f-kube-api-access-zl62d\") pod \"dd384d95-642a-4f20-9981-4dc6eb903c7f\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " Nov 02 18:25:21.899719 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:21.899672 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-home\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-home\") pod \"dd384d95-642a-4f20-9981-4dc6eb903c7f\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " Nov 02 18:25:21.899795 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:21.899774 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-tmp-dir\") pod \"dd384d95-642a-4f20-9981-4dc6eb903c7f\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " Nov 02 18:25:21.899903 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:21.899885 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-model-cache\") pod \"dd384d95-642a-4f20-9981-4dc6eb903c7f\" (UID: \"dd384d95-642a-4f20-9981-4dc6eb903c7f\") " Nov 02 18:25:21.900838 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:21.900093 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-home" pod "dd384d95-642a-4f20-9981-4dc6eb903c7f" (UID: "dd384d95-642a-4f20-9981-4dc6eb903c7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:21.900838 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:21.900200 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-tmp-dir" pod "dd384d95-642a-4f20-9981-4dc6eb903c7f" (UID: "dd384d95-642a-4f20-9981-4dc6eb903c7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:21.900838 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:21.900344 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-model-cache" pod "dd384d95-642a-4f20-9981-4dc6eb903c7f" (UID: "dd384d95-642a-4f20-9981-4dc6eb903c7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:21.900838 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:21.900767 5601 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-home\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:21.900838 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:21.900793 5601 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-tmp-dir\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:21.900838 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:21.900809 5601 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-model-cache\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:21.902274 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:21.902246 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-dshm" pod "dd384d95-642a-4f20-9981-4dc6eb903c7f" (UID: "dd384d95-642a-4f20-9981-4dc6eb903c7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:21.902457 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:21.902440 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd384d95-642a-4f20-9981-4dc6eb903c7f-tls-certs" pod "dd384d95-642a-4f20-9981-4dc6eb903c7f" (UID: "dd384d95-642a-4f20-9981-4dc6eb903c7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:25:21.904003 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:21.903982 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd384d95-642a-4f20-9981-4dc6eb903c7f-kube-api-access-zl62d" pod "dd384d95-642a-4f20-9981-4dc6eb903c7f" (UID: "dd384d95-642a-4f20-9981-4dc6eb903c7f"). InnerVolumeSpecName "kube-api-access-zl62d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:25:22.002367 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:22.002337 5601 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd384d95-642a-4f20-9981-4dc6eb903c7f-tls-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:22.002367 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:22.002363 5601 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd384d95-642a-4f20-9981-4dc6eb903c7f-dshm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:22.002367 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:22.002373 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl62d\" (UniqueName: \"kubernetes.io/projected/dd384d95-642a-4f20-9981-4dc6eb903c7f-kube-api-access-zl62d\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:22.344733 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:22.344690 5601 generic.go:356] "Generic (PLEG): container finished" podID="dd384d95-642a-4f20-9981-4dc6eb903c7f" containerID="5f2492066dbf75ce2e382c3cf5f9fb7549b489831a6bd8b67bf49e17b93adb12" exitCode=0 Nov 02 18:25:22.344966 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:22.344856 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" event={"ID":"dd384d95-642a-4f20-9981-4dc6eb903c7f","Type":"ContainerDied","Data":"5f2492066dbf75ce2e382c3cf5f9fb7549b489831a6bd8b67bf49e17b93adb12"} Nov 02 18:25:22.344966 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:22.344897 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" event={"ID":"dd384d95-642a-4f20-9981-4dc6eb903c7f","Type":"ContainerDied","Data":"5040d10c71b5d792fa2885f252b05455384d8fbac4dd0e5a7336cbed9b96e69a"} Nov 02 18:25:22.344966 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:22.344922 5601 scope.go:122] "RemoveContainer" containerID="5f2492066dbf75ce2e382c3cf5f9fb7549b489831a6bd8b67bf49e17b93adb12" Nov 02 18:25:22.345165 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:22.345101 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7" Nov 02 18:25:22.360226 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:22.360207 5601 scope.go:122] "RemoveContainer" containerID="5f2492066dbf75ce2e382c3cf5f9fb7549b489831a6bd8b67bf49e17b93adb12" Nov 02 18:25:22.360496 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:25:22.360475 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f2492066dbf75ce2e382c3cf5f9fb7549b489831a6bd8b67bf49e17b93adb12\": container with ID starting with 5f2492066dbf75ce2e382c3cf5f9fb7549b489831a6bd8b67bf49e17b93adb12 not found: ID does not exist" containerID="5f2492066dbf75ce2e382c3cf5f9fb7549b489831a6bd8b67bf49e17b93adb12" Nov 02 18:25:22.360556 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:22.360504 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f2492066dbf75ce2e382c3cf5f9fb7549b489831a6bd8b67bf49e17b93adb12"} err="failed to get container status \"5f2492066dbf75ce2e382c3cf5f9fb7549b489831a6bd8b67bf49e17b93adb12\": rpc error: code = NotFound desc = could not find container \"5f2492066dbf75ce2e382c3cf5f9fb7549b489831a6bd8b67bf49e17b93adb12\": container with ID starting with 5f2492066dbf75ce2e382c3cf5f9fb7549b489831a6bd8b67bf49e17b93adb12 not found: ID does not exist" Nov 02 18:25:22.371983 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:22.371958 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7"] Nov 02 18:25:22.375803 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:22.375781 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-50b3a6/test-model-50b3a6-kserve-68644d68d5-rlhr7"] Nov 02 18:25:23.428502 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:23.428469 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd384d95-642a-4f20-9981-4dc6eb903c7f" path="/var/lib/kubelet/pods/dd384d95-642a-4f20-9981-4dc6eb903c7f/volumes" Nov 02 18:25:31.234600 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.234564 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl"] Nov 02 18:25:31.235649 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.235603 5601 cpu_manager.go:414] "containerMap: removing container" podUID="72a481ce-cb98-41e1-bb69-db80124ba5c4" containerName="istio-proxy" Nov 02 18:25:31.235649 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.235642 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a481ce-cb98-41e1-bb69-db80124ba5c4" containerName="istio-proxy" Nov 02 18:25:31.235806 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.235660 5601 cpu_manager.go:414] "containerMap: removing container" podUID="dd384d95-642a-4f20-9981-4dc6eb903c7f" containerName="main" Nov 02 18:25:31.235806 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.235668 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd384d95-642a-4f20-9981-4dc6eb903c7f" containerName="main" Nov 02 18:25:31.235806 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.235696 5601 cpu_manager.go:414] "containerMap: removing container" podUID="3b429f46-034e-43a5-bffb-65f58b9c84a4" containerName="istio-proxy" Nov 02 18:25:31.235806 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.235703 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b429f46-034e-43a5-bffb-65f58b9c84a4" containerName="istio-proxy" Nov 02 18:25:31.235930 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.235856 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a481ce-cb98-41e1-bb69-db80124ba5c4" containerName="istio-proxy" Nov 02 18:25:31.235930 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.235869 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd384d95-642a-4f20-9981-4dc6eb903c7f" containerName="main" Nov 02 18:25:31.235930 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.235882 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b429f46-034e-43a5-bffb-65f58b9c84a4" containerName="istio-proxy" Nov 02 18:25:31.239895 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.239878 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.248851 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.248669 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2966e7-openshift-default-dockercfg-xd7qq\"" Nov 02 18:25:31.253332 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.253310 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl"] Nov 02 18:25:31.338785 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.338749 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istiod-ca-cert\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.338966 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.338796 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-token\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.338966 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.338942 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-envoy\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.339046 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.338987 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-credential-socket\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.339111 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.339100 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-workload-socket\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.339188 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.339171 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9g67\" (UniqueName: \"kubernetes.io/projected/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-kube-api-access-s9g67\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.339252 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.339237 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-podinfo\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.339288 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.339281 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-workload-certs\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.339373 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.339361 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-data\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.443731 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.443693 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-workload-certs\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.444033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.444002 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-data\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.444234 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.444198 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istiod-ca-cert\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.444381 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.444363 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-token\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.444549 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.444532 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-envoy\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.444802 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.444779 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-credential-socket\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.445073 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.445052 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-workload-socket\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.445330 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.445311 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9g67\" (UniqueName: \"kubernetes.io/projected/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-kube-api-access-s9g67\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.445606 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.445574 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-podinfo\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.449915 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.449227 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-podinfo\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.449915 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.449567 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-workload-certs\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.449915 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.449838 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-data\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.450903 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.450810 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istiod-ca-cert\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.458120 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.453836 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-envoy\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.458120 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.454096 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-credential-socket\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.458120 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.454404 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-workload-socket\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.461159 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.460765 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-token\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.463801 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.463768 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9g67\" (UniqueName: \"kubernetes.io/projected/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-kube-api-access-s9g67\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.556254 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.556130 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:31.867396 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.867291 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl"] Nov 02 18:25:31.873826 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:25:31.873784 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9cb412_9cd3_444c_a7d0_ab4e7b397275.slice/crio-07ebb055648a85d443813425c02567b9c175c76320caf04419594024bf297b92 WatchSource:0}: Error finding container 07ebb055648a85d443813425c02567b9c175c76320caf04419594024bf297b92: Status 404 returned error can't find the container with id 07ebb055648a85d443813425c02567b9c175c76320caf04419594024bf297b92 Nov 02 18:25:31.877597 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.877547 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:25:31.877845 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.877816 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:25:31.878011 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:31.877985 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:25:32.479047 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:32.479008 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" event={"ID":"2c9cb412-9cd3-444c-a7d0-ab4e7b397275","Type":"ContainerStarted","Data":"c0d798d006ab8aa2c4a4cbe34b021a3e01fb7fa8b3ce21093f323b8dae0a8ed3"} Nov 02 18:25:32.479481 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:32.479066 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" event={"ID":"2c9cb412-9cd3-444c-a7d0-ab4e7b397275","Type":"ContainerStarted","Data":"07ebb055648a85d443813425c02567b9c175c76320caf04419594024bf297b92"} Nov 02 18:25:32.502659 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:32.499914 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" podStartSLOduration=1.49987524 podStartE2EDuration="1.49987524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:25:32.497721306 +0000 UTC m=+3097.188775317" watchObservedRunningTime="2026-11-02 18:25:32.49987524 +0000 UTC m=+3097.190929252" Nov 02 18:25:32.556597 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:32.556555 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:32.558503 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:32.558476 5601 patch_prober.go:29] interesting pod/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 02 18:25:32.558669 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:32.558524 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" podUID="2c9cb412-9cd3-444c-a7d0-ab4e7b397275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 02 18:25:33.559008 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:33.558964 5601 patch_prober.go:29] interesting pod/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 02 18:25:33.559593 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:33.559047 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" podUID="2c9cb412-9cd3-444c-a7d0-ab4e7b397275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 02 18:25:34.497458 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:25:34.497413 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:25:34.558642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:34.557970 5601 patch_prober.go:29] interesting pod/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 02 18:25:34.558642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:34.558027 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" podUID="2c9cb412-9cd3-444c-a7d0-ab4e7b397275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 02 18:25:35.557640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:35.557056 5601 patch_prober.go:29] interesting pod/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 02 18:25:35.557640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:35.557135 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" podUID="2c9cb412-9cd3-444c-a7d0-ab4e7b397275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 02 18:25:36.558642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:36.557020 5601 patch_prober.go:29] interesting pod/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 02 18:25:36.558642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:36.557086 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" podUID="2c9cb412-9cd3-444c-a7d0-ab4e7b397275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 02 18:25:37.558999 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:37.558962 5601 patch_prober.go:29] interesting pod/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 02 18:25:37.559417 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:37.559030 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" podUID="2c9cb412-9cd3-444c-a7d0-ab4e7b397275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 02 18:25:37.965648 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:37.964959 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw"] Nov 02 18:25:37.979783 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:37.976897 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7"] Nov 02 18:25:38.734300 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:38.734253 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:38.735100 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:38.735074 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:38.736324 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:38.736291 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:25:42.966119 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:42.966080 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw" podUID="c9f29635-35c4-419c-80e4-0239daa2b244" containerName="payload-processing" containerID="cri-o://8e5879bd867f0f4c36aadedb3e27b28d311875648da3fcea03ab1e0ffd0d2e10" gracePeriod=25 Nov 02 18:25:42.978065 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:42.978016 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" podUID="6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed" containerName="payload-pre-processing" containerID="cri-o://e890b62ac95a337fb4cfb9bb42400c0ce1f6136ab92587ac07aa67e3c12f49a2" gracePeriod=25 Nov 02 18:25:43.284405 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:43.284371 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw" Nov 02 18:25:43.329118 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:43.329082 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9f29635-35c4-419c-80e4-0239daa2b244-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f29635-35c4-419c-80e4-0239daa2b244-plugins-config-volume\") pod \"c9f29635-35c4-419c-80e4-0239daa2b244\" (UID: \"c9f29635-35c4-419c-80e4-0239daa2b244\") " Nov 02 18:25:43.329370 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:43.329253 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f29635-35c4-419c-80e4-0239daa2b244-kube-api-access-n44s9\" (UniqueName: \"kubernetes.io/projected/c9f29635-35c4-419c-80e4-0239daa2b244-kube-api-access-n44s9\") pod \"c9f29635-35c4-419c-80e4-0239daa2b244\" (UID: \"c9f29635-35c4-419c-80e4-0239daa2b244\") " Nov 02 18:25:43.329535 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:43.329513 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f29635-35c4-419c-80e4-0239daa2b244-plugins-config-volume" pod "c9f29635-35c4-419c-80e4-0239daa2b244" (UID: "c9f29635-35c4-419c-80e4-0239daa2b244"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:25:43.329957 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:43.329932 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f29635-35c4-419c-80e4-0239daa2b244-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:43.331500 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:43.331472 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f29635-35c4-419c-80e4-0239daa2b244-kube-api-access-n44s9" pod "c9f29635-35c4-419c-80e4-0239daa2b244" (UID: "c9f29635-35c4-419c-80e4-0239daa2b244"). InnerVolumeSpecName "kube-api-access-n44s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:25:43.431371 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:43.431337 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n44s9\" (UniqueName: \"kubernetes.io/projected/c9f29635-35c4-419c-80e4-0239daa2b244-kube-api-access-n44s9\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:43.569495 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:43.569399 5601 generic.go:356] "Generic (PLEG): container finished" podID="c9f29635-35c4-419c-80e4-0239daa2b244" containerID="8e5879bd867f0f4c36aadedb3e27b28d311875648da3fcea03ab1e0ffd0d2e10" exitCode=0 Nov 02 18:25:43.569495 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:43.569457 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw" Nov 02 18:25:43.569690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:43.569499 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw" event={"ID":"c9f29635-35c4-419c-80e4-0239daa2b244","Type":"ContainerDied","Data":"8e5879bd867f0f4c36aadedb3e27b28d311875648da3fcea03ab1e0ffd0d2e10"} Nov 02 18:25:43.569690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:43.569553 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw" event={"ID":"c9f29635-35c4-419c-80e4-0239daa2b244","Type":"ContainerDied","Data":"8f7c0e3b40d12c773165ac24e08e9c8cb029ee382041f915dfc3f384ea5b6796"} Nov 02 18:25:43.569690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:43.569579 5601 scope.go:122] "RemoveContainer" containerID="8e5879bd867f0f4c36aadedb3e27b28d311875648da3fcea03ab1e0ffd0d2e10" Nov 02 18:25:43.581108 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:43.581089 5601 scope.go:122] "RemoveContainer" containerID="8e5879bd867f0f4c36aadedb3e27b28d311875648da3fcea03ab1e0ffd0d2e10" Nov 02 18:25:43.581440 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:25:43.581410 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e5879bd867f0f4c36aadedb3e27b28d311875648da3fcea03ab1e0ffd0d2e10\": container with ID starting with 8e5879bd867f0f4c36aadedb3e27b28d311875648da3fcea03ab1e0ffd0d2e10 not found: ID does not exist" containerID="8e5879bd867f0f4c36aadedb3e27b28d311875648da3fcea03ab1e0ffd0d2e10" Nov 02 18:25:43.581500 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:43.581449 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5879bd867f0f4c36aadedb3e27b28d311875648da3fcea03ab1e0ffd0d2e10"} err="failed to get container status \"8e5879bd867f0f4c36aadedb3e27b28d311875648da3fcea03ab1e0ffd0d2e10\": rpc error: code = NotFound desc = could not find container \"8e5879bd867f0f4c36aadedb3e27b28d311875648da3fcea03ab1e0ffd0d2e10\": container with ID starting with 8e5879bd867f0f4c36aadedb3e27b28d311875648da3fcea03ab1e0ffd0d2e10 not found: ID does not exist" Nov 02 18:25:43.590479 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:43.590455 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw"] Nov 02 18:25:43.595170 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:43.595140 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-06644b-5776d95ff9-zp5tw"] Nov 02 18:25:45.428125 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:45.428086 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9f29635-35c4-419c-80e4-0239daa2b244" path="/var/lib/kubelet/pods/c9f29635-35c4-419c-80e4-0239daa2b244/volumes" Nov 02 18:25:47.193873 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:47.193838 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj"] Nov 02 18:25:47.194693 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:47.194640 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" podUID="ea7a3828-ae1f-4f34-a847-3a03b824cd33" containerName="istio-proxy" containerID="cri-o://c21fe46f43104e7cea2eddc5868ed020b61ddcb1de8aa145813324570d783b23" gracePeriod=30 Nov 02 18:25:48.059779 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:48.059739 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" podUID="6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:25:52.481209 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.481146 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:25:52.617260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.609938 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-credential-socket\") pod \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " Nov 02 18:25:52.617260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.610026 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-envoy\") pod \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " Nov 02 18:25:52.617260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.610094 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-workload-socket\") pod \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " Nov 02 18:25:52.617260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.610229 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-workload-certs\") pod \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " Nov 02 18:25:52.617260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.610359 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-token\" (UniqueName: \"kubernetes.io/projected/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-token\") pod \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " Nov 02 18:25:52.617260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.610514 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istiod-ca-cert\") pod \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " Nov 02 18:25:52.617260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.610541 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-data\") pod \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " Nov 02 18:25:52.617260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.610569 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-podinfo\") pod \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " Nov 02 18:25:52.617260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.610645 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea7a3828-ae1f-4f34-a847-3a03b824cd33-kube-api-access-t6qsr\" (UniqueName: \"kubernetes.io/projected/ea7a3828-ae1f-4f34-a847-3a03b824cd33-kube-api-access-t6qsr\") pod \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\" (UID: \"ea7a3828-ae1f-4f34-a847-3a03b824cd33\") " Nov 02 18:25:52.617260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.613301 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-credential-socket" pod "ea7a3828-ae1f-4f34-a847-3a03b824cd33" (UID: "ea7a3828-ae1f-4f34-a847-3a03b824cd33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:52.618202 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.618001 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-data" pod "ea7a3828-ae1f-4f34-a847-3a03b824cd33" (UID: "ea7a3828-ae1f-4f34-a847-3a03b824cd33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:52.618202 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.618186 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-workload-certs" pod "ea7a3828-ae1f-4f34-a847-3a03b824cd33" (UID: "ea7a3828-ae1f-4f34-a847-3a03b824cd33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:52.621502 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.621441 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istiod-ca-cert" pod "ea7a3828-ae1f-4f34-a847-3a03b824cd33" (UID: "ea7a3828-ae1f-4f34-a847-3a03b824cd33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:25:52.622105 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.622082 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-workload-socket" pod "ea7a3828-ae1f-4f34-a847-3a03b824cd33" (UID: "ea7a3828-ae1f-4f34-a847-3a03b824cd33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:52.622759 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.622721 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7a3828-ae1f-4f34-a847-3a03b824cd33-kube-api-access-t6qsr" pod "ea7a3828-ae1f-4f34-a847-3a03b824cd33" (UID: "ea7a3828-ae1f-4f34-a847-3a03b824cd33"). InnerVolumeSpecName "kube-api-access-t6qsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:25:52.640372 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.639413 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-envoy" pod "ea7a3828-ae1f-4f34-a847-3a03b824cd33" (UID: "ea7a3828-ae1f-4f34-a847-3a03b824cd33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:52.640511 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.640393 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-token" pod "ea7a3828-ae1f-4f34-a847-3a03b824cd33" (UID: "ea7a3828-ae1f-4f34-a847-3a03b824cd33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:25:52.641698 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.641670 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-podinfo" pod "ea7a3828-ae1f-4f34-a847-3a03b824cd33" (UID: "ea7a3828-ae1f-4f34-a847-3a03b824cd33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:25:52.663887 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.663850 5601 generic.go:356] "Generic (PLEG): container finished" podID="ea7a3828-ae1f-4f34-a847-3a03b824cd33" containerID="c21fe46f43104e7cea2eddc5868ed020b61ddcb1de8aa145813324570d783b23" exitCode=0 Nov 02 18:25:52.664075 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.663907 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" Nov 02 18:25:52.664075 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.663934 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" event={"ID":"ea7a3828-ae1f-4f34-a847-3a03b824cd33","Type":"ContainerDied","Data":"c21fe46f43104e7cea2eddc5868ed020b61ddcb1de8aa145813324570d783b23"} Nov 02 18:25:52.664075 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.664002 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj" event={"ID":"ea7a3828-ae1f-4f34-a847-3a03b824cd33","Type":"ContainerDied","Data":"46af90cc9bf9d712abc6965328810fbc5db4f83a7f40988d3f6d174577cd458a"} Nov 02 18:25:52.664075 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.664036 5601 scope.go:122] "RemoveContainer" containerID="c21fe46f43104e7cea2eddc5868ed020b61ddcb1de8aa145813324570d783b23" Nov 02 18:25:52.675940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.675920 5601 scope.go:122] "RemoveContainer" containerID="a047143d12c6314cf0662fbff8e7c329aced046d5435f2e7e0c0bb62585ea600" Nov 02 18:25:52.686382 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.686362 5601 scope.go:122] "RemoveContainer" containerID="c21fe46f43104e7cea2eddc5868ed020b61ddcb1de8aa145813324570d783b23" Nov 02 18:25:52.686683 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:25:52.686658 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c21fe46f43104e7cea2eddc5868ed020b61ddcb1de8aa145813324570d783b23\": container with ID starting with c21fe46f43104e7cea2eddc5868ed020b61ddcb1de8aa145813324570d783b23 not found: ID does not exist" containerID="c21fe46f43104e7cea2eddc5868ed020b61ddcb1de8aa145813324570d783b23" Nov 02 18:25:52.686818 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.686693 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21fe46f43104e7cea2eddc5868ed020b61ddcb1de8aa145813324570d783b23"} err="failed to get container status \"c21fe46f43104e7cea2eddc5868ed020b61ddcb1de8aa145813324570d783b23\": rpc error: code = NotFound desc = could not find container \"c21fe46f43104e7cea2eddc5868ed020b61ddcb1de8aa145813324570d783b23\": container with ID starting with c21fe46f43104e7cea2eddc5868ed020b61ddcb1de8aa145813324570d783b23 not found: ID does not exist" Nov 02 18:25:52.686818 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.686720 5601 scope.go:122] "RemoveContainer" containerID="a047143d12c6314cf0662fbff8e7c329aced046d5435f2e7e0c0bb62585ea600" Nov 02 18:25:52.686969 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:25:52.686951 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a047143d12c6314cf0662fbff8e7c329aced046d5435f2e7e0c0bb62585ea600\": container with ID starting with a047143d12c6314cf0662fbff8e7c329aced046d5435f2e7e0c0bb62585ea600 not found: ID does not exist" containerID="a047143d12c6314cf0662fbff8e7c329aced046d5435f2e7e0c0bb62585ea600" Nov 02 18:25:52.687015 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.686976 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a047143d12c6314cf0662fbff8e7c329aced046d5435f2e7e0c0bb62585ea600"} err="failed to get container status \"a047143d12c6314cf0662fbff8e7c329aced046d5435f2e7e0c0bb62585ea600\": rpc error: code = NotFound desc = could not find container \"a047143d12c6314cf0662fbff8e7c329aced046d5435f2e7e0c0bb62585ea600\": container with ID starting with a047143d12c6314cf0662fbff8e7c329aced046d5435f2e7e0c0bb62585ea600 not found: ID does not exist" Nov 02 18:25:52.693506 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.693477 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj"] Nov 02 18:25:52.697043 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.697017 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-06644b-openshift-default-f5fcd9d8f-x4zmj"] Nov 02 18:25:52.713460 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.713428 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:52.713460 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.713458 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:52.713758 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.713470 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:52.713758 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.713480 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:52.713758 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.713488 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:52.713758 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.713496 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:52.713758 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.713506 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:52.713758 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.713518 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7a3828-ae1f-4f34-a847-3a03b824cd33-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:52.713758 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:52.713527 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6qsr\" (UniqueName: \"kubernetes.io/projected/ea7a3828-ae1f-4f34-a847-3a03b824cd33-kube-api-access-t6qsr\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:25:53.427425 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:53.427391 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea7a3828-ae1f-4f34-a847-3a03b824cd33" path="/var/lib/kubelet/pods/ea7a3828-ae1f-4f34-a847-3a03b824cd33/volumes" Nov 02 18:25:58.058996 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:25:58.058958 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" podUID="6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:26:03.307065 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:26:03.307034 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" Nov 02 18:26:03.419436 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:26:03.419396 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed-kube-api-access-8pflb\" (UniqueName: \"kubernetes.io/projected/6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed-kube-api-access-8pflb\") pod \"6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed\" (UID: \"6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed\") " Nov 02 18:26:03.419658 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:26:03.419464 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed-plugins-config-volume\") pod \"6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed\" (UID: \"6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed\") " Nov 02 18:26:03.419855 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:26:03.419832 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed-plugins-config-volume" pod "6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed" (UID: "6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:26:03.420103 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:26:03.420072 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:26:03.421750 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:26:03.421720 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed-kube-api-access-8pflb" pod "6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed" (UID: "6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed"). InnerVolumeSpecName "kube-api-access-8pflb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:26:03.522102 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:26:03.522016 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pflb\" (UniqueName: \"kubernetes.io/projected/6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed-kube-api-access-8pflb\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:26:03.764439 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:26:03.764404 5601 generic.go:356] "Generic (PLEG): container finished" podID="6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed" containerID="e890b62ac95a337fb4cfb9bb42400c0ce1f6136ab92587ac07aa67e3c12f49a2" exitCode=0 Nov 02 18:26:03.764651 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:26:03.764446 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" Nov 02 18:26:03.764651 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:26:03.764489 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" event={"ID":"6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed","Type":"ContainerDied","Data":"e890b62ac95a337fb4cfb9bb42400c0ce1f6136ab92587ac07aa67e3c12f49a2"} Nov 02 18:26:03.764651 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:26:03.764544 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7" event={"ID":"6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed","Type":"ContainerDied","Data":"63230317c4100296e202b9822ebd61de595ba274462d61c54ff149fb4b511b23"} Nov 02 18:26:03.764651 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:26:03.764573 5601 scope.go:122] "RemoveContainer" containerID="e890b62ac95a337fb4cfb9bb42400c0ce1f6136ab92587ac07aa67e3c12f49a2" Nov 02 18:26:03.778168 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:26:03.778125 5601 scope.go:122] "RemoveContainer" containerID="e890b62ac95a337fb4cfb9bb42400c0ce1f6136ab92587ac07aa67e3c12f49a2" Nov 02 18:26:03.778709 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:26:03.778684 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e890b62ac95a337fb4cfb9bb42400c0ce1f6136ab92587ac07aa67e3c12f49a2\": container with ID starting with e890b62ac95a337fb4cfb9bb42400c0ce1f6136ab92587ac07aa67e3c12f49a2 not found: ID does not exist" containerID="e890b62ac95a337fb4cfb9bb42400c0ce1f6136ab92587ac07aa67e3c12f49a2" Nov 02 18:26:03.778861 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:26:03.778836 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e890b62ac95a337fb4cfb9bb42400c0ce1f6136ab92587ac07aa67e3c12f49a2"} err="failed to get container status \"e890b62ac95a337fb4cfb9bb42400c0ce1f6136ab92587ac07aa67e3c12f49a2\": rpc error: code = NotFound desc = could not find container \"e890b62ac95a337fb4cfb9bb42400c0ce1f6136ab92587ac07aa67e3c12f49a2\": container with ID starting with e890b62ac95a337fb4cfb9bb42400c0ce1f6136ab92587ac07aa67e3c12f49a2 not found: ID does not exist" Nov 02 18:26:03.788253 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:26:03.788221 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7"] Nov 02 18:26:03.791876 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:26:03.791853 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-06644b-7c6db78bb7-bcxb7"] Nov 02 18:26:04.486034 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:26:04.485994 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:26:05.428057 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:26:05.428011 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed" path="/var/lib/kubelet/pods/6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed/volumes" Nov 02 18:26:34.485540 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:26:34.485496 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:27:04.486104 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:27:04.486064 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:27:33.561828 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:33.561791 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8"] Nov 02 18:27:34.495510 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:27:34.495449 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:27:48.584632 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:48.584576 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" podUID="6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f" containerName="main" containerID="cri-o://a87c32e75680034091af1161730fdeaf8af7bf2513e5aee74b912fb56093a52a" gracePeriod=45 Nov 02 18:27:48.884840 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:48.884810 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:27:48.976295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:48.976267 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-model-cache\") pod \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " Nov 02 18:27:48.976295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:48.976302 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-tmp-dir\") pod \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " Nov 02 18:27:48.976582 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:48.976336 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-dshm\") pod \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " Nov 02 18:27:48.976582 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:48.976419 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-kube-api-access-dl992\" (UniqueName: \"kubernetes.io/projected/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-kube-api-access-dl992\") pod \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " Nov 02 18:27:48.976582 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:48.976461 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-tls-certs\") pod \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " Nov 02 18:27:48.976582 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:48.976563 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-model-cache" pod "6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f" (UID: "6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:48.976833 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:48.976579 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-tmp-dir" pod "6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f" (UID: "6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:48.976833 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:48.976674 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-home\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-home\") pod \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\" (UID: \"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f\") " Nov 02 18:27:48.977011 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:48.976986 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-home" pod "6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f" (UID: "6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:48.977258 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:48.977236 5601 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-home\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:27:48.977327 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:48.977259 5601 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-model-cache\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:27:48.977327 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:48.977272 5601 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-tmp-dir\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:27:48.978976 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:48.978946 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-kube-api-access-dl992" pod "6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f" (UID: "6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f"). InnerVolumeSpecName "kube-api-access-dl992". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:27:48.979093 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:48.979056 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-dshm" pod "6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f" (UID: "6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:48.979093 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:48.979073 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-tls-certs" pod "6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f" (UID: "6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:27:49.079563 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:49.079027 5601 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-tls-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:27:49.079563 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:49.079065 5601 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-dshm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:27:49.079563 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:49.079084 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl992\" (UniqueName: \"kubernetes.io/projected/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f-kube-api-access-dl992\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:27:49.118682 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:49.110209 5601 generic.go:356] "Generic (PLEG): container finished" podID="6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f" containerID="a87c32e75680034091af1161730fdeaf8af7bf2513e5aee74b912fb56093a52a" exitCode=0 Nov 02 18:27:49.118682 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:49.110452 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" event={"ID":"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f","Type":"ContainerDied","Data":"a87c32e75680034091af1161730fdeaf8af7bf2513e5aee74b912fb56093a52a"} Nov 02 18:27:49.118682 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:49.110500 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" event={"ID":"6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f","Type":"ContainerDied","Data":"b605734a9f8826c23132bd598c6d84dc1a1379ae5801955f337cab44f89b400a"} Nov 02 18:27:49.118682 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:49.110536 5601 scope.go:122] "RemoveContainer" containerID="a87c32e75680034091af1161730fdeaf8af7bf2513e5aee74b912fb56093a52a" Nov 02 18:27:49.118682 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:49.110745 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8" Nov 02 18:27:49.129643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:49.129602 5601 scope.go:122] "RemoveContainer" containerID="a87c32e75680034091af1161730fdeaf8af7bf2513e5aee74b912fb56093a52a" Nov 02 18:27:49.129911 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:27:49.129891 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87c32e75680034091af1161730fdeaf8af7bf2513e5aee74b912fb56093a52a\": container with ID starting with a87c32e75680034091af1161730fdeaf8af7bf2513e5aee74b912fb56093a52a not found: ID does not exist" containerID="a87c32e75680034091af1161730fdeaf8af7bf2513e5aee74b912fb56093a52a" Nov 02 18:27:49.129979 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:49.129920 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87c32e75680034091af1161730fdeaf8af7bf2513e5aee74b912fb56093a52a"} err="failed to get container status \"a87c32e75680034091af1161730fdeaf8af7bf2513e5aee74b912fb56093a52a\": rpc error: code = NotFound desc = could not find container \"a87c32e75680034091af1161730fdeaf8af7bf2513e5aee74b912fb56093a52a\": container with ID starting with a87c32e75680034091af1161730fdeaf8af7bf2513e5aee74b912fb56093a52a not found: ID does not exist" Nov 02 18:27:49.141226 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:49.141202 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8"] Nov 02 18:27:49.145291 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:49.145261 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2966e7/test-model-2966e7-kserve-896499f7c-lfxs8"] Nov 02 18:27:49.432971 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:49.430486 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f" path="/var/lib/kubelet/pods/6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f/volumes" Nov 02 18:27:51.618523 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:51.618486 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m"] Nov 02 18:27:56.619075 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:27:56.619034 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" podUID="993ea19b-d958-4efa-91c6-e9ba381b23ef" containerName="payload-processing" containerID="cri-o://4dea66ae88b5b4f49c2bc8f3068c23e70f6e4c0491a97ce8e6eb6d5bb8b537ed" gracePeriod=25 Nov 02 18:28:00.380775 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:00.380738 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" podUID="993ea19b-d958-4efa-91c6-e9ba381b23ef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:28:04.486931 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:28:04.486893 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:28:06.507641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:06.505172 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl"] Nov 02 18:28:06.507641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:06.505396 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" podUID="2c9cb412-9cd3-444c-a7d0-ab4e7b397275" containerName="istio-proxy" containerID="cri-o://c0d798d006ab8aa2c4a4cbe34b021a3e01fb7fa8b3ce21093f323b8dae0a8ed3" gracePeriod=30 Nov 02 18:28:08.737484 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:08.737401 5601 patch_prober.go:29] interesting pod/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:28:08.737484 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:08.737457 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" podUID="2c9cb412-9cd3-444c-a7d0-ab4e7b397275" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:28:10.380873 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:10.380836 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" podUID="993ea19b-d958-4efa-91c6-e9ba381b23ef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:28:11.807179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.807150 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:28:11.909127 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.909092 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-credential-socket\") pod \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " Nov 02 18:28:11.909327 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.909161 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-envoy\") pod \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " Nov 02 18:28:11.909327 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.909217 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-kube-api-access-s9g67\" (UniqueName: \"kubernetes.io/projected/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-kube-api-access-s9g67\") pod \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " Nov 02 18:28:11.909327 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.909307 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-token\" (UniqueName: \"kubernetes.io/projected/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-token\") pod \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " Nov 02 18:28:11.909523 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.909347 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-workload-socket\") pod \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " Nov 02 18:28:11.909523 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.909394 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-workload-certs\") pod \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " Nov 02 18:28:11.909523 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.909495 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-credential-socket" pod "2c9cb412-9cd3-444c-a7d0-ab4e7b397275" (UID: "2c9cb412-9cd3-444c-a7d0-ab4e7b397275"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:11.909715 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.909553 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istiod-ca-cert\") pod \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " Nov 02 18:28:11.909715 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.909586 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-podinfo\") pod \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " Nov 02 18:28:11.909715 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.909689 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-data\") pod \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\" (UID: \"2c9cb412-9cd3-444c-a7d0-ab4e7b397275\") " Nov 02 18:28:11.909888 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.909806 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-workload-certs" pod "2c9cb412-9cd3-444c-a7d0-ab4e7b397275" (UID: "2c9cb412-9cd3-444c-a7d0-ab4e7b397275"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:11.909957 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.909908 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-workload-socket" pod "2c9cb412-9cd3-444c-a7d0-ab4e7b397275" (UID: "2c9cb412-9cd3-444c-a7d0-ab4e7b397275"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:11.910145 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.910121 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istiod-ca-cert" pod "2c9cb412-9cd3-444c-a7d0-ab4e7b397275" (UID: "2c9cb412-9cd3-444c-a7d0-ab4e7b397275"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:11.910422 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.910402 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-data" pod "2c9cb412-9cd3-444c-a7d0-ab4e7b397275" (UID: "2c9cb412-9cd3-444c-a7d0-ab4e7b397275"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:11.910690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.910669 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:11.910781 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.910693 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:11.910781 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.910707 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:11.910781 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.910721 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:11.910781 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.910734 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:11.911999 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.911971 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-envoy" pod "2c9cb412-9cd3-444c-a7d0-ab4e7b397275" (UID: "2c9cb412-9cd3-444c-a7d0-ab4e7b397275"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:11.912260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.912239 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-kube-api-access-s9g67" pod "2c9cb412-9cd3-444c-a7d0-ab4e7b397275" (UID: "2c9cb412-9cd3-444c-a7d0-ab4e7b397275"). InnerVolumeSpecName "kube-api-access-s9g67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:11.912495 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.912470 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-token" pod "2c9cb412-9cd3-444c-a7d0-ab4e7b397275" (UID: "2c9cb412-9cd3-444c-a7d0-ab4e7b397275"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:11.912732 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:11.912711 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-podinfo" pod "2c9cb412-9cd3-444c-a7d0-ab4e7b397275" (UID: "2c9cb412-9cd3-444c-a7d0-ab4e7b397275"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:28:12.013136 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:12.013100 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:12.013136 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:12.013130 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9g67\" (UniqueName: \"kubernetes.io/projected/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-kube-api-access-s9g67\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:12.013136 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:12.013140 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:12.013378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:12.013148 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9cb412-9cd3-444c-a7d0-ab4e7b397275-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:12.376249 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:12.376204 5601 generic.go:356] "Generic (PLEG): container finished" podID="2c9cb412-9cd3-444c-a7d0-ab4e7b397275" containerID="c0d798d006ab8aa2c4a4cbe34b021a3e01fb7fa8b3ce21093f323b8dae0a8ed3" exitCode=0 Nov 02 18:28:12.376249 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:12.376245 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" Nov 02 18:28:12.376541 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:12.376286 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" event={"ID":"2c9cb412-9cd3-444c-a7d0-ab4e7b397275","Type":"ContainerDied","Data":"c0d798d006ab8aa2c4a4cbe34b021a3e01fb7fa8b3ce21093f323b8dae0a8ed3"} Nov 02 18:28:12.376541 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:12.376352 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl" event={"ID":"2c9cb412-9cd3-444c-a7d0-ab4e7b397275","Type":"ContainerDied","Data":"07ebb055648a85d443813425c02567b9c175c76320caf04419594024bf297b92"} Nov 02 18:28:12.376541 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:12.376385 5601 scope.go:122] "RemoveContainer" containerID="c0d798d006ab8aa2c4a4cbe34b021a3e01fb7fa8b3ce21093f323b8dae0a8ed3" Nov 02 18:28:12.390712 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:12.390688 5601 scope.go:122] "RemoveContainer" containerID="c0d798d006ab8aa2c4a4cbe34b021a3e01fb7fa8b3ce21093f323b8dae0a8ed3" Nov 02 18:28:12.391003 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:28:12.390986 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d798d006ab8aa2c4a4cbe34b021a3e01fb7fa8b3ce21093f323b8dae0a8ed3\": container with ID starting with c0d798d006ab8aa2c4a4cbe34b021a3e01fb7fa8b3ce21093f323b8dae0a8ed3 not found: ID does not exist" containerID="c0d798d006ab8aa2c4a4cbe34b021a3e01fb7fa8b3ce21093f323b8dae0a8ed3" Nov 02 18:28:12.391073 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:12.391009 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d798d006ab8aa2c4a4cbe34b021a3e01fb7fa8b3ce21093f323b8dae0a8ed3"} err="failed to get container status \"c0d798d006ab8aa2c4a4cbe34b021a3e01fb7fa8b3ce21093f323b8dae0a8ed3\": rpc error: code = NotFound desc = could not find container \"c0d798d006ab8aa2c4a4cbe34b021a3e01fb7fa8b3ce21093f323b8dae0a8ed3\": container with ID starting with c0d798d006ab8aa2c4a4cbe34b021a3e01fb7fa8b3ce21093f323b8dae0a8ed3 not found: ID does not exist" Nov 02 18:28:12.404316 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:12.404283 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl"] Nov 02 18:28:12.407828 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:12.407802 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-rg7hl"] Nov 02 18:28:13.427384 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:13.427348 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c9cb412-9cd3-444c-a7d0-ab4e7b397275" path="/var/lib/kubelet/pods/2c9cb412-9cd3-444c-a7d0-ab4e7b397275/volumes" Nov 02 18:28:16.903440 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:16.903416 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" Nov 02 18:28:16.996355 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:16.996325 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993ea19b-d958-4efa-91c6-e9ba381b23ef-kube-api-access-ln6g8\" (UniqueName: \"kubernetes.io/projected/993ea19b-d958-4efa-91c6-e9ba381b23ef-kube-api-access-ln6g8\") pod \"993ea19b-d958-4efa-91c6-e9ba381b23ef\" (UID: \"993ea19b-d958-4efa-91c6-e9ba381b23ef\") " Nov 02 18:28:16.996528 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:16.996515 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993ea19b-d958-4efa-91c6-e9ba381b23ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993ea19b-d958-4efa-91c6-e9ba381b23ef-plugins-config-volume\") pod \"993ea19b-d958-4efa-91c6-e9ba381b23ef\" (UID: \"993ea19b-d958-4efa-91c6-e9ba381b23ef\") " Nov 02 18:28:16.996978 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:16.996945 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993ea19b-d958-4efa-91c6-e9ba381b23ef-plugins-config-volume" pod "993ea19b-d958-4efa-91c6-e9ba381b23ef" (UID: "993ea19b-d958-4efa-91c6-e9ba381b23ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:16.998813 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:16.998778 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993ea19b-d958-4efa-91c6-e9ba381b23ef-kube-api-access-ln6g8" pod "993ea19b-d958-4efa-91c6-e9ba381b23ef" (UID: "993ea19b-d958-4efa-91c6-e9ba381b23ef"). InnerVolumeSpecName "kube-api-access-ln6g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:17.098654 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:17.098604 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993ea19b-d958-4efa-91c6-e9ba381b23ef-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:17.098654 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:17.098650 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln6g8\" (UniqueName: \"kubernetes.io/projected/993ea19b-d958-4efa-91c6-e9ba381b23ef-kube-api-access-ln6g8\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:17.416997 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:17.416966 5601 generic.go:356] "Generic (PLEG): container finished" podID="993ea19b-d958-4efa-91c6-e9ba381b23ef" containerID="4dea66ae88b5b4f49c2bc8f3068c23e70f6e4c0491a97ce8e6eb6d5bb8b537ed" exitCode=0 Nov 02 18:28:17.417204 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:17.417019 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" Nov 02 18:28:17.417204 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:17.417054 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" event={"ID":"993ea19b-d958-4efa-91c6-e9ba381b23ef","Type":"ContainerDied","Data":"4dea66ae88b5b4f49c2bc8f3068c23e70f6e4c0491a97ce8e6eb6d5bb8b537ed"} Nov 02 18:28:17.417204 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:17.417109 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m" event={"ID":"993ea19b-d958-4efa-91c6-e9ba381b23ef","Type":"ContainerDied","Data":"f4a60ebf8aa17b950ca82913466084ca84ecaa8b86f73799713003fbdf78ed68"} Nov 02 18:28:17.417204 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:17.417139 5601 scope.go:122] "RemoveContainer" containerID="4dea66ae88b5b4f49c2bc8f3068c23e70f6e4c0491a97ce8e6eb6d5bb8b537ed" Nov 02 18:28:17.429896 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:17.429872 5601 scope.go:122] "RemoveContainer" containerID="4dea66ae88b5b4f49c2bc8f3068c23e70f6e4c0491a97ce8e6eb6d5bb8b537ed" Nov 02 18:28:17.430166 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:28:17.430142 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dea66ae88b5b4f49c2bc8f3068c23e70f6e4c0491a97ce8e6eb6d5bb8b537ed\": container with ID starting with 4dea66ae88b5b4f49c2bc8f3068c23e70f6e4c0491a97ce8e6eb6d5bb8b537ed not found: ID does not exist" containerID="4dea66ae88b5b4f49c2bc8f3068c23e70f6e4c0491a97ce8e6eb6d5bb8b537ed" Nov 02 18:28:17.430228 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:17.430176 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dea66ae88b5b4f49c2bc8f3068c23e70f6e4c0491a97ce8e6eb6d5bb8b537ed"} err="failed to get container status \"4dea66ae88b5b4f49c2bc8f3068c23e70f6e4c0491a97ce8e6eb6d5bb8b537ed\": rpc error: code = NotFound desc = could not find container \"4dea66ae88b5b4f49c2bc8f3068c23e70f6e4c0491a97ce8e6eb6d5bb8b537ed\": container with ID starting with 4dea66ae88b5b4f49c2bc8f3068c23e70f6e4c0491a97ce8e6eb6d5bb8b537ed not found: ID does not exist" Nov 02 18:28:17.445354 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:17.445321 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m"] Nov 02 18:28:17.450586 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:17.450556 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2966e7-6d8c7c698c-jgf2m"] Nov 02 18:28:18.343114 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:18.343077 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m"] Nov 02 18:28:18.344017 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:18.343983 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" podUID="7adb9d7e-827f-4573-a294-2681a27a79e0" containerName="maas-api" containerID="cri-o://80e93daee116eb265fc28f553827d2074a9b84a20340589bc420202a50e520b8" gracePeriod=30 Nov 02 18:28:18.932012 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:18.931973 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:28:18.999322 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:18.999289 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-metrics-certs\") pod \"7adb9d7e-827f-4573-a294-2681a27a79e0\" (UID: \"7adb9d7e-827f-4573-a294-2681a27a79e0\") " Nov 02 18:28:18.999527 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:18.999489 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7adb9d7e-827f-4573-a294-2681a27a79e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7adb9d7e-827f-4573-a294-2681a27a79e0-openshift-service-ca\") pod \"7adb9d7e-827f-4573-a294-2681a27a79e0\" (UID: \"7adb9d7e-827f-4573-a294-2681a27a79e0\") " Nov 02 18:28:18.999594 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:18.999540 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7adb9d7e-827f-4573-a294-2681a27a79e0-kube-api-access-fgqwm\" (UniqueName: \"kubernetes.io/projected/7adb9d7e-827f-4573-a294-2681a27a79e0-kube-api-access-fgqwm\") pod \"7adb9d7e-827f-4573-a294-2681a27a79e0\" (UID: \"7adb9d7e-827f-4573-a294-2681a27a79e0\") " Nov 02 18:28:18.999732 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:18.999708 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-maas-api-tls\") pod \"7adb9d7e-827f-4573-a294-2681a27a79e0\" (UID: \"7adb9d7e-827f-4573-a294-2681a27a79e0\") " Nov 02 18:28:18.999905 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:18.999882 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7adb9d7e-827f-4573-a294-2681a27a79e0-openshift-service-ca" pod "7adb9d7e-827f-4573-a294-2681a27a79e0" (UID: "7adb9d7e-827f-4573-a294-2681a27a79e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:19.000352 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:19.000329 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7adb9d7e-827f-4573-a294-2681a27a79e0-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:19.001876 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:19.001854 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-maas-api-tls" pod "7adb9d7e-827f-4573-a294-2681a27a79e0" (UID: "7adb9d7e-827f-4573-a294-2681a27a79e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:28:19.002120 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:19.002092 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7adb9d7e-827f-4573-a294-2681a27a79e0-kube-api-access-fgqwm" pod "7adb9d7e-827f-4573-a294-2681a27a79e0" (UID: "7adb9d7e-827f-4573-a294-2681a27a79e0"). InnerVolumeSpecName "kube-api-access-fgqwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:19.002251 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:19.002092 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-metrics-certs" pod "7adb9d7e-827f-4573-a294-2681a27a79e0" (UID: "7adb9d7e-827f-4573-a294-2681a27a79e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:28:19.101643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:19.101574 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:19.101643 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:19.101608 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgqwm\" (UniqueName: \"kubernetes.io/projected/7adb9d7e-827f-4573-a294-2681a27a79e0-kube-api-access-fgqwm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:19.101852 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:19.101654 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7adb9d7e-827f-4573-a294-2681a27a79e0-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:19.429457 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:19.429412 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993ea19b-d958-4efa-91c6-e9ba381b23ef" path="/var/lib/kubelet/pods/993ea19b-d958-4efa-91c6-e9ba381b23ef/volumes" Nov 02 18:28:19.436649 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:19.436582 5601 generic.go:356] "Generic (PLEG): container finished" podID="7adb9d7e-827f-4573-a294-2681a27a79e0" containerID="80e93daee116eb265fc28f553827d2074a9b84a20340589bc420202a50e520b8" exitCode=0 Nov 02 18:28:19.436836 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:19.436650 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" Nov 02 18:28:19.436836 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:19.436658 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" event={"ID":"7adb9d7e-827f-4573-a294-2681a27a79e0","Type":"ContainerDied","Data":"80e93daee116eb265fc28f553827d2074a9b84a20340589bc420202a50e520b8"} Nov 02 18:28:19.436836 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:19.436725 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m" event={"ID":"7adb9d7e-827f-4573-a294-2681a27a79e0","Type":"ContainerDied","Data":"0cac2eca335dca4ba5d9bed998eff96d9fea048837841bb1db5cdf9e271e5461"} Nov 02 18:28:19.436836 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:19.436756 5601 scope.go:122] "RemoveContainer" containerID="80e93daee116eb265fc28f553827d2074a9b84a20340589bc420202a50e520b8" Nov 02 18:28:19.451179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:19.451159 5601 scope.go:122] "RemoveContainer" containerID="80e93daee116eb265fc28f553827d2074a9b84a20340589bc420202a50e520b8" Nov 02 18:28:19.451512 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:28:19.451487 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e93daee116eb265fc28f553827d2074a9b84a20340589bc420202a50e520b8\": container with ID starting with 80e93daee116eb265fc28f553827d2074a9b84a20340589bc420202a50e520b8 not found: ID does not exist" containerID="80e93daee116eb265fc28f553827d2074a9b84a20340589bc420202a50e520b8" Nov 02 18:28:19.451565 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:19.451525 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e93daee116eb265fc28f553827d2074a9b84a20340589bc420202a50e520b8"} err="failed to get container status \"80e93daee116eb265fc28f553827d2074a9b84a20340589bc420202a50e520b8\": rpc error: code = NotFound desc = could not find container \"80e93daee116eb265fc28f553827d2074a9b84a20340589bc420202a50e520b8\": container with ID starting with 80e93daee116eb265fc28f553827d2074a9b84a20340589bc420202a50e520b8 not found: ID does not exist" Nov 02 18:28:19.458819 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:19.458788 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m"] Nov 02 18:28:19.462093 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:19.462070 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-807c08-df65db78d-wgf8m"] Nov 02 18:28:21.427714 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:21.427681 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7adb9d7e-827f-4573-a294-2681a27a79e0" path="/var/lib/kubelet/pods/7adb9d7e-827f-4573-a294-2681a27a79e0/volumes" Nov 02 18:28:34.484863 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:28:34.484823 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:28:46.142952 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:46.142914 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm"] Nov 02 18:28:46.143796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:46.143760 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" podUID="145f6bf1-b596-4e1e-a561-10c46c7c9b63" containerName="maas-api" containerID="cri-o://f7be535fd735982751bec3196dfe6fa59fa2a5973661aab9ef4987587b2ae5e6" gracePeriod=30 Nov 02 18:28:46.656662 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:46.656611 5601 generic.go:356] "Generic (PLEG): container finished" podID="145f6bf1-b596-4e1e-a561-10c46c7c9b63" containerID="f7be535fd735982751bec3196dfe6fa59fa2a5973661aab9ef4987587b2ae5e6" exitCode=0 Nov 02 18:28:46.656799 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:46.656688 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" event={"ID":"145f6bf1-b596-4e1e-a561-10c46c7c9b63","Type":"ContainerDied","Data":"f7be535fd735982751bec3196dfe6fa59fa2a5973661aab9ef4987587b2ae5e6"} Nov 02 18:28:46.656799 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:46.656746 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" event={"ID":"145f6bf1-b596-4e1e-a561-10c46c7c9b63","Type":"ContainerDied","Data":"ba48037596af70caf7908e736f5054b5219fe0d09581d62a98c5ab1f9f8d18e2"} Nov 02 18:28:46.656799 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:46.656763 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba48037596af70caf7908e736f5054b5219fe0d09581d62a98c5ab1f9f8d18e2" Nov 02 18:28:46.678900 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:46.678878 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" Nov 02 18:28:46.832149 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:46.832109 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/145f6bf1-b596-4e1e-a561-10c46c7c9b63-metrics-certs\" (UniqueName: \"kubernetes.io/secret/145f6bf1-b596-4e1e-a561-10c46c7c9b63-metrics-certs\") pod \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\" (UID: \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\") " Nov 02 18:28:46.832576 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:46.832549 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/145f6bf1-b596-4e1e-a561-10c46c7c9b63-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145f6bf1-b596-4e1e-a561-10c46c7c9b63-maas-api-tls\") pod \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\" (UID: \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\") " Nov 02 18:28:46.832845 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:46.832736 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145f6bf1-b596-4e1e-a561-10c46c7c9b63-kube-api-access-hdsq4\" (UniqueName: \"kubernetes.io/projected/145f6bf1-b596-4e1e-a561-10c46c7c9b63-kube-api-access-hdsq4\") pod \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\" (UID: \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\") " Nov 02 18:28:46.833073 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:46.833045 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/145f6bf1-b596-4e1e-a561-10c46c7c9b63-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145f6bf1-b596-4e1e-a561-10c46c7c9b63-openshift-service-ca\") pod \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\" (UID: \"145f6bf1-b596-4e1e-a561-10c46c7c9b63\") " Nov 02 18:28:46.835549 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:46.835522 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145f6bf1-b596-4e1e-a561-10c46c7c9b63-openshift-service-ca" pod "145f6bf1-b596-4e1e-a561-10c46c7c9b63" (UID: "145f6bf1-b596-4e1e-a561-10c46c7c9b63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:46.847647 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:46.847094 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145f6bf1-b596-4e1e-a561-10c46c7c9b63-metrics-certs" pod "145f6bf1-b596-4e1e-a561-10c46c7c9b63" (UID: "145f6bf1-b596-4e1e-a561-10c46c7c9b63"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:28:46.848116 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:46.848076 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145f6bf1-b596-4e1e-a561-10c46c7c9b63-maas-api-tls" pod "145f6bf1-b596-4e1e-a561-10c46c7c9b63" (UID: "145f6bf1-b596-4e1e-a561-10c46c7c9b63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:28:46.850608 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:46.850581 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145f6bf1-b596-4e1e-a561-10c46c7c9b63-kube-api-access-hdsq4" pod "145f6bf1-b596-4e1e-a561-10c46c7c9b63" (UID: "145f6bf1-b596-4e1e-a561-10c46c7c9b63"). InnerVolumeSpecName "kube-api-access-hdsq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:46.936333 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:46.936298 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145f6bf1-b596-4e1e-a561-10c46c7c9b63-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:46.936333 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:46.936327 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145f6bf1-b596-4e1e-a561-10c46c7c9b63-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:46.936333 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:46.936338 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdsq4\" (UniqueName: \"kubernetes.io/projected/145f6bf1-b596-4e1e-a561-10c46c7c9b63-kube-api-access-hdsq4\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:46.936574 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:46.936349 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145f6bf1-b596-4e1e-a561-10c46c7c9b63-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:47.347809 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:47.339731 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj"] Nov 02 18:28:47.347809 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:47.339974 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" podUID="d8fa2df6-995c-4f63-b100-8dcec94085d7" containerName="istio-proxy" containerID="cri-o://f7c8a6edae06d9381e228bcadcc83b412f957343d048d7feabfd127af48321dd" gracePeriod=30 Nov 02 18:28:47.664820 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:47.664730 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm" Nov 02 18:28:47.683390 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:47.683338 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm"] Nov 02 18:28:47.686751 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:47.686726 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bee8f6-8f5d8b7b5-mgzpm"] Nov 02 18:28:49.427682 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:49.427644 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="145f6bf1-b596-4e1e-a561-10c46c7c9b63" path="/var/lib/kubelet/pods/145f6bf1-b596-4e1e-a561-10c46c7c9b63/volumes" Nov 02 18:28:52.893082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.841037 5601 generic.go:356] "Generic (PLEG): container finished" podID="d8fa2df6-995c-4f63-b100-8dcec94085d7" containerID="f7c8a6edae06d9381e228bcadcc83b412f957343d048d7feabfd127af48321dd" exitCode=0 Nov 02 18:28:52.893082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.841148 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" event={"ID":"d8fa2df6-995c-4f63-b100-8dcec94085d7","Type":"ContainerDied","Data":"f7c8a6edae06d9381e228bcadcc83b412f957343d048d7feabfd127af48321dd"} Nov 02 18:28:52.893082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.846565 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:28:52.893082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.871107 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8fa2df6-995c-4f63-b100-8dcec94085d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fa2df6-995c-4f63-b100-8dcec94085d7-istiod-ca-cert\") pod \"d8fa2df6-995c-4f63-b100-8dcec94085d7\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " Nov 02 18:28:52.893082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.871291 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-token\" (UniqueName: \"kubernetes.io/projected/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-token\") pod \"d8fa2df6-995c-4f63-b100-8dcec94085d7\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " Nov 02 18:28:52.893082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.871358 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-credential-socket\") pod \"d8fa2df6-995c-4f63-b100-8dcec94085d7\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " Nov 02 18:28:52.893082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.871411 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-data\") pod \"d8fa2df6-995c-4f63-b100-8dcec94085d7\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " Nov 02 18:28:52.893082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.871474 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-workload-certs\") pod \"d8fa2df6-995c-4f63-b100-8dcec94085d7\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " Nov 02 18:28:52.893082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.871526 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8fa2df6-995c-4f63-b100-8dcec94085d7-kube-api-access-9qr7m\" (UniqueName: \"kubernetes.io/projected/d8fa2df6-995c-4f63-b100-8dcec94085d7-kube-api-access-9qr7m\") pod \"d8fa2df6-995c-4f63-b100-8dcec94085d7\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " Nov 02 18:28:52.893082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.885304 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-token" pod "d8fa2df6-995c-4f63-b100-8dcec94085d7" (UID: "d8fa2df6-995c-4f63-b100-8dcec94085d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:52.893082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.885729 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8fa2df6-995c-4f63-b100-8dcec94085d7-istiod-ca-cert" pod "d8fa2df6-995c-4f63-b100-8dcec94085d7" (UID: "d8fa2df6-995c-4f63-b100-8dcec94085d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:52.893082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.885975 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-data" pod "d8fa2df6-995c-4f63-b100-8dcec94085d7" (UID: "d8fa2df6-995c-4f63-b100-8dcec94085d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:52.893082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.886143 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-credential-socket" pod "d8fa2df6-995c-4f63-b100-8dcec94085d7" (UID: "d8fa2df6-995c-4f63-b100-8dcec94085d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:52.893082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.886326 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-workload-certs" pod "d8fa2df6-995c-4f63-b100-8dcec94085d7" (UID: "d8fa2df6-995c-4f63-b100-8dcec94085d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:52.894292 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.893483 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8fa2df6-995c-4f63-b100-8dcec94085d7-kube-api-access-9qr7m" pod "d8fa2df6-995c-4f63-b100-8dcec94085d7" (UID: "d8fa2df6-995c-4f63-b100-8dcec94085d7"). InnerVolumeSpecName "kube-api-access-9qr7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:52.926396 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.901649 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-workload-socket\") pod \"d8fa2df6-995c-4f63-b100-8dcec94085d7\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " Nov 02 18:28:52.926396 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.901709 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-envoy\") pod \"d8fa2df6-995c-4f63-b100-8dcec94085d7\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " Nov 02 18:28:52.926396 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.901755 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-podinfo\") pod \"d8fa2df6-995c-4f63-b100-8dcec94085d7\" (UID: \"d8fa2df6-995c-4f63-b100-8dcec94085d7\") " Nov 02 18:28:52.926396 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.902512 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:52.926396 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.902534 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:52.926396 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.902548 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:52.926396 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.902563 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:52.926396 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.902576 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qr7m\" (UniqueName: \"kubernetes.io/projected/d8fa2df6-995c-4f63-b100-8dcec94085d7-kube-api-access-9qr7m\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:52.926396 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.902590 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fa2df6-995c-4f63-b100-8dcec94085d7-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:52.926396 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.917303 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-workload-socket" pod "d8fa2df6-995c-4f63-b100-8dcec94085d7" (UID: "d8fa2df6-995c-4f63-b100-8dcec94085d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:52.926396 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.922292 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-podinfo" pod "d8fa2df6-995c-4f63-b100-8dcec94085d7" (UID: "d8fa2df6-995c-4f63-b100-8dcec94085d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:28:52.926396 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:52.923849 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-envoy" pod "d8fa2df6-995c-4f63-b100-8dcec94085d7" (UID: "d8fa2df6-995c-4f63-b100-8dcec94085d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:53.004503 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:53.004424 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:53.004503 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:53.004455 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:53.004503 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:53.004465 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fa2df6-995c-4f63-b100-8dcec94085d7-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:53.851889 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:53.851849 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" Nov 02 18:28:53.852099 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:53.851846 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj" event={"ID":"d8fa2df6-995c-4f63-b100-8dcec94085d7","Type":"ContainerDied","Data":"0636116adcc2488cba2ecd5a0256e13f632b82a326b435dd1e6cb68358733056"} Nov 02 18:28:53.852099 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:53.852000 5601 scope.go:122] "RemoveContainer" containerID="f7c8a6edae06d9381e228bcadcc83b412f957343d048d7feabfd127af48321dd" Nov 02 18:28:53.874837 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:53.874806 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj"] Nov 02 18:28:53.877036 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:53.877009 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lwtbj"] Nov 02 18:28:55.454920 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:55.454878 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8fa2df6-995c-4f63-b100-8dcec94085d7" path="/var/lib/kubelet/pods/d8fa2df6-995c-4f63-b100-8dcec94085d7/volumes" Nov 02 18:28:57.255255 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.255219 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc"] Nov 02 18:28:57.256097 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.256040 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" podUID="b98ac353-914e-4f05-b8b5-2c29eb3703d0" containerName="maas-api" containerID="cri-o://1fadfd30e93738c4f3b3e7cbe965ae45ee619224e3dddd65a415e433f11b3080" gracePeriod=30 Nov 02 18:28:57.592811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.592709 5601 scope.go:122] "RemoveContainer" containerID="f7be535fd735982751bec3196dfe6fa59fa2a5973661aab9ef4987587b2ae5e6" Nov 02 18:28:57.766948 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.766923 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" Nov 02 18:28:57.889571 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.889465 5601 generic.go:356] "Generic (PLEG): container finished" podID="b98ac353-914e-4f05-b8b5-2c29eb3703d0" containerID="1fadfd30e93738c4f3b3e7cbe965ae45ee619224e3dddd65a415e433f11b3080" exitCode=0 Nov 02 18:28:57.889571 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.889514 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" Nov 02 18:28:57.889571 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.889545 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" event={"ID":"b98ac353-914e-4f05-b8b5-2c29eb3703d0","Type":"ContainerDied","Data":"1fadfd30e93738c4f3b3e7cbe965ae45ee619224e3dddd65a415e433f11b3080"} Nov 02 18:28:57.889880 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.889603 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc" event={"ID":"b98ac353-914e-4f05-b8b5-2c29eb3703d0","Type":"ContainerDied","Data":"f393778ba06f734c084fe880fd4128e3bd6dc360fed34a490953574259da4452"} Nov 02 18:28:57.889880 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.889629 5601 scope.go:122] "RemoveContainer" containerID="1fadfd30e93738c4f3b3e7cbe965ae45ee619224e3dddd65a415e433f11b3080" Nov 02 18:28:57.901260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.901240 5601 scope.go:122] "RemoveContainer" containerID="1fadfd30e93738c4f3b3e7cbe965ae45ee619224e3dddd65a415e433f11b3080" Nov 02 18:28:57.901546 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:28:57.901528 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fadfd30e93738c4f3b3e7cbe965ae45ee619224e3dddd65a415e433f11b3080\": container with ID starting with 1fadfd30e93738c4f3b3e7cbe965ae45ee619224e3dddd65a415e433f11b3080 not found: ID does not exist" containerID="1fadfd30e93738c4f3b3e7cbe965ae45ee619224e3dddd65a415e433f11b3080" Nov 02 18:28:57.901661 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.901556 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fadfd30e93738c4f3b3e7cbe965ae45ee619224e3dddd65a415e433f11b3080"} err="failed to get container status \"1fadfd30e93738c4f3b3e7cbe965ae45ee619224e3dddd65a415e433f11b3080\": rpc error: code = NotFound desc = could not find container \"1fadfd30e93738c4f3b3e7cbe965ae45ee619224e3dddd65a415e433f11b3080\": container with ID starting with 1fadfd30e93738c4f3b3e7cbe965ae45ee619224e3dddd65a415e433f11b3080 not found: ID does not exist" Nov 02 18:28:57.934246 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.934217 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b98ac353-914e-4f05-b8b5-2c29eb3703d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98ac353-914e-4f05-b8b5-2c29eb3703d0-maas-api-tls\") pod \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\" (UID: \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\") " Nov 02 18:28:57.934361 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.934345 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b98ac353-914e-4f05-b8b5-2c29eb3703d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98ac353-914e-4f05-b8b5-2c29eb3703d0-metrics-certs\") pod \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\" (UID: \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\") " Nov 02 18:28:57.934411 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.934389 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b98ac353-914e-4f05-b8b5-2c29eb3703d0-kube-api-access-445gg\" (UniqueName: \"kubernetes.io/projected/b98ac353-914e-4f05-b8b5-2c29eb3703d0-kube-api-access-445gg\") pod \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\" (UID: \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\") " Nov 02 18:28:57.934447 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.934434 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b98ac353-914e-4f05-b8b5-2c29eb3703d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98ac353-914e-4f05-b8b5-2c29eb3703d0-openshift-service-ca\") pod \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\" (UID: \"b98ac353-914e-4f05-b8b5-2c29eb3703d0\") " Nov 02 18:28:57.934892 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.934866 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98ac353-914e-4f05-b8b5-2c29eb3703d0-openshift-service-ca" pod "b98ac353-914e-4f05-b8b5-2c29eb3703d0" (UID: "b98ac353-914e-4f05-b8b5-2c29eb3703d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:57.935144 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.935118 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98ac353-914e-4f05-b8b5-2c29eb3703d0-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:57.936517 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.936487 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98ac353-914e-4f05-b8b5-2c29eb3703d0-metrics-certs" pod "b98ac353-914e-4f05-b8b5-2c29eb3703d0" (UID: "b98ac353-914e-4f05-b8b5-2c29eb3703d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:28:57.936692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.936530 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98ac353-914e-4f05-b8b5-2c29eb3703d0-maas-api-tls" pod "b98ac353-914e-4f05-b8b5-2c29eb3703d0" (UID: "b98ac353-914e-4f05-b8b5-2c29eb3703d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:28:57.936692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:57.936581 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98ac353-914e-4f05-b8b5-2c29eb3703d0-kube-api-access-445gg" pod "b98ac353-914e-4f05-b8b5-2c29eb3703d0" (UID: "b98ac353-914e-4f05-b8b5-2c29eb3703d0"). InnerVolumeSpecName "kube-api-access-445gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:58.041364 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:58.036771 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98ac353-914e-4f05-b8b5-2c29eb3703d0-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:58.041364 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:58.036809 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-445gg\" (UniqueName: \"kubernetes.io/projected/b98ac353-914e-4f05-b8b5-2c29eb3703d0-kube-api-access-445gg\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:58.041364 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:58.036825 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98ac353-914e-4f05-b8b5-2c29eb3703d0-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:28:58.222637 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:58.219965 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc"] Nov 02 18:28:58.226774 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:58.225812 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b6b6f4-6b889749df-c5hnc"] Nov 02 18:28:58.231461 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:58.231430 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc"] Nov 02 18:28:58.231715 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:58.231676 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" podUID="8fd72fbc-016a-43de-95f0-4d939b864aac" containerName="istio-proxy" containerID="cri-o://1cd6d046ceb47e2e820dcd8d41d0bdc2f247a635962d75f60256b950e1b09ff5" gracePeriod=30 Nov 02 18:28:59.427149 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:28:59.427112 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b98ac353-914e-4f05-b8b5-2c29eb3703d0" path="/var/lib/kubelet/pods/b98ac353-914e-4f05-b8b5-2c29eb3703d0/volumes" Nov 02 18:29:02.093480 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:02.093429 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r"] Nov 02 18:29:02.094000 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:02.093686 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" podUID="dc1c481d-b715-4ffd-8803-7371e21eead0" containerName="istio-proxy" containerID="cri-o://ccbc8fe3bf80234c11ddfbd335950e0c8faaf9b9e845352f1514aa5bc936428a" gracePeriod=30 Nov 02 18:29:03.568205 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.568169 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:29:03.654109 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.654067 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-workload-socket\") pod \"8fd72fbc-016a-43de-95f0-4d939b864aac\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " Nov 02 18:29:03.654299 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.654122 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fd72fbc-016a-43de-95f0-4d939b864aac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd72fbc-016a-43de-95f0-4d939b864aac-istiod-ca-cert\") pod \"8fd72fbc-016a-43de-95f0-4d939b864aac\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " Nov 02 18:29:03.654299 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.654170 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-token\" (UniqueName: \"kubernetes.io/projected/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-token\") pod \"8fd72fbc-016a-43de-95f0-4d939b864aac\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " Nov 02 18:29:03.654299 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.654231 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fd72fbc-016a-43de-95f0-4d939b864aac-kube-api-access-cpd4c\" (UniqueName: \"kubernetes.io/projected/8fd72fbc-016a-43de-95f0-4d939b864aac-kube-api-access-cpd4c\") pod \"8fd72fbc-016a-43de-95f0-4d939b864aac\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " Nov 02 18:29:03.654478 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.654359 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-envoy\") pod \"8fd72fbc-016a-43de-95f0-4d939b864aac\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " Nov 02 18:29:03.654478 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.654369 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-workload-socket" pod "8fd72fbc-016a-43de-95f0-4d939b864aac" (UID: "8fd72fbc-016a-43de-95f0-4d939b864aac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:03.654478 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.654436 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-podinfo\") pod \"8fd72fbc-016a-43de-95f0-4d939b864aac\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " Nov 02 18:29:03.654680 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.654565 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-workload-certs\") pod \"8fd72fbc-016a-43de-95f0-4d939b864aac\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " Nov 02 18:29:03.654680 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.654601 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd72fbc-016a-43de-95f0-4d939b864aac-istiod-ca-cert" pod "8fd72fbc-016a-43de-95f0-4d939b864aac" (UID: "8fd72fbc-016a-43de-95f0-4d939b864aac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:03.654680 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.654669 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-data\") pod \"8fd72fbc-016a-43de-95f0-4d939b864aac\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " Nov 02 18:29:03.654865 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.654700 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-credential-socket\") pod \"8fd72fbc-016a-43de-95f0-4d939b864aac\" (UID: \"8fd72fbc-016a-43de-95f0-4d939b864aac\") " Nov 02 18:29:03.654865 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.654844 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-workload-certs" pod "8fd72fbc-016a-43de-95f0-4d939b864aac" (UID: "8fd72fbc-016a-43de-95f0-4d939b864aac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:03.655018 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.654992 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-data" pod "8fd72fbc-016a-43de-95f0-4d939b864aac" (UID: "8fd72fbc-016a-43de-95f0-4d939b864aac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:03.655165 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.655144 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-credential-socket" pod "8fd72fbc-016a-43de-95f0-4d939b864aac" (UID: "8fd72fbc-016a-43de-95f0-4d939b864aac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:03.655361 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.655343 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:03.655433 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.655367 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:03.655433 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.655382 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:03.655433 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.655396 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:03.655433 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.655411 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd72fbc-016a-43de-95f0-4d939b864aac-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:03.656716 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.656692 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd72fbc-016a-43de-95f0-4d939b864aac-kube-api-access-cpd4c" pod "8fd72fbc-016a-43de-95f0-4d939b864aac" (UID: "8fd72fbc-016a-43de-95f0-4d939b864aac"). InnerVolumeSpecName "kube-api-access-cpd4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:03.657123 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.657099 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-token" pod "8fd72fbc-016a-43de-95f0-4d939b864aac" (UID: "8fd72fbc-016a-43de-95f0-4d939b864aac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:03.657203 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.657183 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-podinfo" pod "8fd72fbc-016a-43de-95f0-4d939b864aac" (UID: "8fd72fbc-016a-43de-95f0-4d939b864aac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:29:03.657347 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.657325 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-envoy" pod "8fd72fbc-016a-43de-95f0-4d939b864aac" (UID: "8fd72fbc-016a-43de-95f0-4d939b864aac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:03.756383 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.756345 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:03.756383 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.756377 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:03.756383 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.756387 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd72fbc-016a-43de-95f0-4d939b864aac-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:03.756753 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.756397 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpd4c\" (UniqueName: \"kubernetes.io/projected/8fd72fbc-016a-43de-95f0-4d939b864aac-kube-api-access-cpd4c\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:03.951082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.951038 5601 generic.go:356] "Generic (PLEG): container finished" podID="8fd72fbc-016a-43de-95f0-4d939b864aac" containerID="1cd6d046ceb47e2e820dcd8d41d0bdc2f247a635962d75f60256b950e1b09ff5" exitCode=0 Nov 02 18:29:03.951306 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.951092 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" Nov 02 18:29:03.951306 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.951125 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" event={"ID":"8fd72fbc-016a-43de-95f0-4d939b864aac","Type":"ContainerDied","Data":"1cd6d046ceb47e2e820dcd8d41d0bdc2f247a635962d75f60256b950e1b09ff5"} Nov 02 18:29:03.951424 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.951303 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc" event={"ID":"8fd72fbc-016a-43de-95f0-4d939b864aac","Type":"ContainerDied","Data":"1a78c2b521ddb62cf496943013605dec09fe2e4142c781d6a159f4e8a4f649dd"} Nov 02 18:29:03.951424 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.951331 5601 scope.go:122] "RemoveContainer" containerID="1cd6d046ceb47e2e820dcd8d41d0bdc2f247a635962d75f60256b950e1b09ff5" Nov 02 18:29:03.965497 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.965461 5601 scope.go:122] "RemoveContainer" containerID="1cd6d046ceb47e2e820dcd8d41d0bdc2f247a635962d75f60256b950e1b09ff5" Nov 02 18:29:03.965988 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:29:03.965964 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd6d046ceb47e2e820dcd8d41d0bdc2f247a635962d75f60256b950e1b09ff5\": container with ID starting with 1cd6d046ceb47e2e820dcd8d41d0bdc2f247a635962d75f60256b950e1b09ff5 not found: ID does not exist" containerID="1cd6d046ceb47e2e820dcd8d41d0bdc2f247a635962d75f60256b950e1b09ff5" Nov 02 18:29:03.966086 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.965996 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd6d046ceb47e2e820dcd8d41d0bdc2f247a635962d75f60256b950e1b09ff5"} err="failed to get container status \"1cd6d046ceb47e2e820dcd8d41d0bdc2f247a635962d75f60256b950e1b09ff5\": rpc error: code = NotFound desc = could not find container \"1cd6d046ceb47e2e820dcd8d41d0bdc2f247a635962d75f60256b950e1b09ff5\": container with ID starting with 1cd6d046ceb47e2e820dcd8d41d0bdc2f247a635962d75f60256b950e1b09ff5 not found: ID does not exist" Nov 02 18:29:03.980665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.980593 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc"] Nov 02 18:29:03.985191 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:03.985164 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-9c6bc"] Nov 02 18:29:04.485210 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:29:04.485173 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:29:05.446641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:05.446500 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fd72fbc-016a-43de-95f0-4d939b864aac" path="/var/lib/kubelet/pods/8fd72fbc-016a-43de-95f0-4d939b864aac/volumes" Nov 02 18:29:05.451752 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:05.451591 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:29:05.451752 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:05.451739 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" podUID="dc1c481d-b715-4ffd-8803-7371e21eead0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:29:07.403028 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.403001 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 18:29:07.458823 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.458782 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-workload-socket\") pod \"dc1c481d-b715-4ffd-8803-7371e21eead0\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " Nov 02 18:29:07.459025 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.458856 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1c481d-b715-4ffd-8803-7371e21eead0-kube-api-access-xgbls\" (UniqueName: \"kubernetes.io/projected/dc1c481d-b715-4ffd-8803-7371e21eead0-kube-api-access-xgbls\") pod \"dc1c481d-b715-4ffd-8803-7371e21eead0\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " Nov 02 18:29:07.459025 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.458890 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-data\") pod \"dc1c481d-b715-4ffd-8803-7371e21eead0\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " Nov 02 18:29:07.459025 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.458911 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-envoy\") pod \"dc1c481d-b715-4ffd-8803-7371e21eead0\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " Nov 02 18:29:07.459025 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.458955 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-podinfo\") pod \"dc1c481d-b715-4ffd-8803-7371e21eead0\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " Nov 02 18:29:07.459025 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.458999 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-workload-certs\") pod \"dc1c481d-b715-4ffd-8803-7371e21eead0\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " Nov 02 18:29:07.459316 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.459030 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-workload-socket" pod "dc1c481d-b715-4ffd-8803-7371e21eead0" (UID: "dc1c481d-b715-4ffd-8803-7371e21eead0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:07.459316 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.459053 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-token\" (UniqueName: \"kubernetes.io/projected/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-token\") pod \"dc1c481d-b715-4ffd-8803-7371e21eead0\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " Nov 02 18:29:07.459316 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.459109 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-credential-socket\") pod \"dc1c481d-b715-4ffd-8803-7371e21eead0\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " Nov 02 18:29:07.459316 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.459166 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc1c481d-b715-4ffd-8803-7371e21eead0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc1c481d-b715-4ffd-8803-7371e21eead0-istiod-ca-cert\") pod \"dc1c481d-b715-4ffd-8803-7371e21eead0\" (UID: \"dc1c481d-b715-4ffd-8803-7371e21eead0\") " Nov 02 18:29:07.459316 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.459280 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-data" pod "dc1c481d-b715-4ffd-8803-7371e21eead0" (UID: "dc1c481d-b715-4ffd-8803-7371e21eead0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:07.459585 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.459402 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-credential-socket" pod "dc1c481d-b715-4ffd-8803-7371e21eead0" (UID: "dc1c481d-b715-4ffd-8803-7371e21eead0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:07.459677 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.459652 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc1c481d-b715-4ffd-8803-7371e21eead0-istiod-ca-cert" pod "dc1c481d-b715-4ffd-8803-7371e21eead0" (UID: "dc1c481d-b715-4ffd-8803-7371e21eead0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:07.460110 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.459985 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:07.460110 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.460013 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:07.460110 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.460034 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:07.460110 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.460049 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc1c481d-b715-4ffd-8803-7371e21eead0-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:07.460488 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.460256 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-workload-certs" pod "dc1c481d-b715-4ffd-8803-7371e21eead0" (UID: "dc1c481d-b715-4ffd-8803-7371e21eead0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:07.461848 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.461810 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-token" pod "dc1c481d-b715-4ffd-8803-7371e21eead0" (UID: "dc1c481d-b715-4ffd-8803-7371e21eead0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:07.461954 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.461881 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-podinfo" pod "dc1c481d-b715-4ffd-8803-7371e21eead0" (UID: "dc1c481d-b715-4ffd-8803-7371e21eead0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:29:07.462127 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.462095 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-envoy" pod "dc1c481d-b715-4ffd-8803-7371e21eead0" (UID: "dc1c481d-b715-4ffd-8803-7371e21eead0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:07.462248 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.462167 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1c481d-b715-4ffd-8803-7371e21eead0-kube-api-access-xgbls" pod "dc1c481d-b715-4ffd-8803-7371e21eead0" (UID: "dc1c481d-b715-4ffd-8803-7371e21eead0"). InnerVolumeSpecName "kube-api-access-xgbls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:07.561689 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.561651 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgbls\" (UniqueName: \"kubernetes.io/projected/dc1c481d-b715-4ffd-8803-7371e21eead0-kube-api-access-xgbls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:07.561689 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.561685 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:07.561689 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.561695 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:07.561938 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.561704 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc1c481d-b715-4ffd-8803-7371e21eead0-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:07.561938 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:07.561713 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc1c481d-b715-4ffd-8803-7371e21eead0-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:08.049932 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:08.039399 5601 generic.go:356] "Generic (PLEG): container finished" podID="dc1c481d-b715-4ffd-8803-7371e21eead0" containerID="ccbc8fe3bf80234c11ddfbd335950e0c8faaf9b9e845352f1514aa5bc936428a" exitCode=0 Nov 02 18:29:08.049932 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:08.039549 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" event={"ID":"dc1c481d-b715-4ffd-8803-7371e21eead0","Type":"ContainerDied","Data":"ccbc8fe3bf80234c11ddfbd335950e0c8faaf9b9e845352f1514aa5bc936428a"} Nov 02 18:29:08.049932 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:08.039601 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" event={"ID":"dc1c481d-b715-4ffd-8803-7371e21eead0","Type":"ContainerDied","Data":"37094bd7fcf2b0dd6dbe3b06792e79ced12ce078441c8b307468056b089e16f1"} Nov 02 18:29:08.049932 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:08.039659 5601 scope.go:122] "RemoveContainer" containerID="ccbc8fe3bf80234c11ddfbd335950e0c8faaf9b9e845352f1514aa5bc936428a" Nov 02 18:29:08.049932 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:08.039828 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r" Nov 02 18:29:08.074857 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:08.074824 5601 scope.go:122] "RemoveContainer" containerID="ccbc8fe3bf80234c11ddfbd335950e0c8faaf9b9e845352f1514aa5bc936428a" Nov 02 18:29:08.075171 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:29:08.075145 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccbc8fe3bf80234c11ddfbd335950e0c8faaf9b9e845352f1514aa5bc936428a\": container with ID starting with ccbc8fe3bf80234c11ddfbd335950e0c8faaf9b9e845352f1514aa5bc936428a not found: ID does not exist" containerID="ccbc8fe3bf80234c11ddfbd335950e0c8faaf9b9e845352f1514aa5bc936428a" Nov 02 18:29:08.075266 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:08.075178 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccbc8fe3bf80234c11ddfbd335950e0c8faaf9b9e845352f1514aa5bc936428a"} err="failed to get container status \"ccbc8fe3bf80234c11ddfbd335950e0c8faaf9b9e845352f1514aa5bc936428a\": rpc error: code = NotFound desc = could not find container \"ccbc8fe3bf80234c11ddfbd335950e0c8faaf9b9e845352f1514aa5bc936428a\": container with ID starting with ccbc8fe3bf80234c11ddfbd335950e0c8faaf9b9e845352f1514aa5bc936428a not found: ID does not exist" Nov 02 18:29:08.084052 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:08.084025 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r"] Nov 02 18:29:08.089674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:08.089643 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-vwn2r"] Nov 02 18:29:09.427211 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:09.427177 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc1c481d-b715-4ffd-8803-7371e21eead0" path="/var/lib/kubelet/pods/dc1c481d-b715-4ffd-8803-7371e21eead0/volumes" Nov 02 18:29:14.050324 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:14.050287 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t"] Nov 02 18:29:14.050858 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:14.050544 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" podUID="5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2" containerName="maas-api" containerID="cri-o://de6d837ef31d7b7e3d22ef58192e15ee382da2f69737daafef445f7107ff0644" gracePeriod=30 Nov 02 18:29:14.524214 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:14.524175 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" Nov 02 18:29:14.625108 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:14.625079 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-maas-api-tls\") pod \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\" (UID: \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\") " Nov 02 18:29:14.625287 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:14.625120 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-metrics-certs\") pod \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\" (UID: \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\") " Nov 02 18:29:14.625287 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:14.625149 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-kube-api-access-blhwc\" (UniqueName: \"kubernetes.io/projected/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-kube-api-access-blhwc\") pod \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\" (UID: \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\") " Nov 02 18:29:14.625287 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:14.625227 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-openshift-service-ca\") pod \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\" (UID: \"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2\") " Nov 02 18:29:14.626132 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:14.625871 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-openshift-service-ca" pod "5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2" (UID: "5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:14.627223 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:14.627197 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-maas-api-tls" pod "5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2" (UID: "5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:14.627346 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:14.627293 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-metrics-certs" pod "5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2" (UID: "5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:14.627402 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:14.627377 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-kube-api-access-blhwc" pod "5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2" (UID: "5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2"). InnerVolumeSpecName "kube-api-access-blhwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:14.726999 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:14.726965 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:14.726999 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:14.726997 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:14.727190 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:14.727008 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blhwc\" (UniqueName: \"kubernetes.io/projected/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-kube-api-access-blhwc\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:14.727190 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:14.727019 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:15.097498 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:15.097466 5601 generic.go:356] "Generic (PLEG): container finished" podID="5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2" containerID="de6d837ef31d7b7e3d22ef58192e15ee382da2f69737daafef445f7107ff0644" exitCode=0 Nov 02 18:29:15.097977 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:15.097524 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" Nov 02 18:29:15.097977 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:15.097558 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" event={"ID":"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2","Type":"ContainerDied","Data":"de6d837ef31d7b7e3d22ef58192e15ee382da2f69737daafef445f7107ff0644"} Nov 02 18:29:15.097977 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:15.097633 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t" event={"ID":"5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2","Type":"ContainerDied","Data":"038ee623bf1ade4a321abc486a356f9cdb435bcf937d1055e5a1970e546a3c8f"} Nov 02 18:29:15.097977 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:15.097669 5601 scope.go:122] "RemoveContainer" containerID="de6d837ef31d7b7e3d22ef58192e15ee382da2f69737daafef445f7107ff0644" Nov 02 18:29:15.110543 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:15.110505 5601 scope.go:122] "RemoveContainer" containerID="de6d837ef31d7b7e3d22ef58192e15ee382da2f69737daafef445f7107ff0644" Nov 02 18:29:15.110917 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:29:15.110884 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6d837ef31d7b7e3d22ef58192e15ee382da2f69737daafef445f7107ff0644\": container with ID starting with de6d837ef31d7b7e3d22ef58192e15ee382da2f69737daafef445f7107ff0644 not found: ID does not exist" containerID="de6d837ef31d7b7e3d22ef58192e15ee382da2f69737daafef445f7107ff0644" Nov 02 18:29:15.111013 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:15.110920 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6d837ef31d7b7e3d22ef58192e15ee382da2f69737daafef445f7107ff0644"} err="failed to get container status \"de6d837ef31d7b7e3d22ef58192e15ee382da2f69737daafef445f7107ff0644\": rpc error: code = NotFound desc = could not find container \"de6d837ef31d7b7e3d22ef58192e15ee382da2f69737daafef445f7107ff0644\": container with ID starting with de6d837ef31d7b7e3d22ef58192e15ee382da2f69737daafef445f7107ff0644 not found: ID does not exist" Nov 02 18:29:15.122709 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:15.122686 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t"] Nov 02 18:29:15.125942 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:15.125913 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-327608-55b6ffd6fc-4lq8t"] Nov 02 18:29:15.430470 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:15.430432 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2" path="/var/lib/kubelet/pods/5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2/volumes" Nov 02 18:29:20.661916 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:20.661878 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9"] Nov 02 18:29:20.662782 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:20.662744 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" podUID="575ac171-83b1-4d32-bc19-9a039f885c0a" containerName="istio-proxy" containerID="cri-o://f71922eb9dcdf9562dce8036c729ff9ba3fc3d85fb32951c45edacf28b8ddb8f" gracePeriod=30 Nov 02 18:29:23.719712 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:23.719657 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl"] Nov 02 18:29:23.720269 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:23.719926 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" podUID="0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" containerName="istio-proxy" containerID="cri-o://84c9171fa2cacf21d0e7db8324e5f6ff5a09e7c2938af1e60bb3b39ddacf01d5" gracePeriod=30 Nov 02 18:29:25.945568 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:25.945504 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:29:26.045743 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.045705 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-token\" (UniqueName: \"kubernetes.io/projected/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-token\") pod \"575ac171-83b1-4d32-bc19-9a039f885c0a\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " Nov 02 18:29:26.045914 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.045801 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/575ac171-83b1-4d32-bc19-9a039f885c0a-kube-api-access-bf577\" (UniqueName: \"kubernetes.io/projected/575ac171-83b1-4d32-bc19-9a039f885c0a-kube-api-access-bf577\") pod \"575ac171-83b1-4d32-bc19-9a039f885c0a\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " Nov 02 18:29:26.045914 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.045837 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-credential-socket\") pod \"575ac171-83b1-4d32-bc19-9a039f885c0a\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " Nov 02 18:29:26.045914 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.045888 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-podinfo\") pod \"575ac171-83b1-4d32-bc19-9a039f885c0a\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " Nov 02 18:29:26.046118 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.045948 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-data\") pod \"575ac171-83b1-4d32-bc19-9a039f885c0a\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " Nov 02 18:29:26.046118 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.046008 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-workload-socket\") pod \"575ac171-83b1-4d32-bc19-9a039f885c0a\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " Nov 02 18:29:26.046118 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.046056 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/575ac171-83b1-4d32-bc19-9a039f885c0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/575ac171-83b1-4d32-bc19-9a039f885c0a-istiod-ca-cert\") pod \"575ac171-83b1-4d32-bc19-9a039f885c0a\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " Nov 02 18:29:26.046284 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.046165 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-envoy\") pod \"575ac171-83b1-4d32-bc19-9a039f885c0a\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " Nov 02 18:29:26.046284 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.046250 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-workload-certs\") pod \"575ac171-83b1-4d32-bc19-9a039f885c0a\" (UID: \"575ac171-83b1-4d32-bc19-9a039f885c0a\") " Nov 02 18:29:26.046385 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.046283 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-data" pod "575ac171-83b1-4d32-bc19-9a039f885c0a" (UID: "575ac171-83b1-4d32-bc19-9a039f885c0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:26.046442 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.046370 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-workload-socket" pod "575ac171-83b1-4d32-bc19-9a039f885c0a" (UID: "575ac171-83b1-4d32-bc19-9a039f885c0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:26.046503 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.046487 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-credential-socket" pod "575ac171-83b1-4d32-bc19-9a039f885c0a" (UID: "575ac171-83b1-4d32-bc19-9a039f885c0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:26.046667 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.046645 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-workload-certs" pod "575ac171-83b1-4d32-bc19-9a039f885c0a" (UID: "575ac171-83b1-4d32-bc19-9a039f885c0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:26.047577 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.047000 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:26.047577 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.047026 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:26.047577 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.047069 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:26.047908 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.046647 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/575ac171-83b1-4d32-bc19-9a039f885c0a-istiod-ca-cert" pod "575ac171-83b1-4d32-bc19-9a039f885c0a" (UID: "575ac171-83b1-4d32-bc19-9a039f885c0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:26.048828 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.048802 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-token" pod "575ac171-83b1-4d32-bc19-9a039f885c0a" (UID: "575ac171-83b1-4d32-bc19-9a039f885c0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:26.048939 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.048837 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575ac171-83b1-4d32-bc19-9a039f885c0a-kube-api-access-bf577" pod "575ac171-83b1-4d32-bc19-9a039f885c0a" (UID: "575ac171-83b1-4d32-bc19-9a039f885c0a"). InnerVolumeSpecName "kube-api-access-bf577". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:26.048990 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.048941 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-podinfo" pod "575ac171-83b1-4d32-bc19-9a039f885c0a" (UID: "575ac171-83b1-4d32-bc19-9a039f885c0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:29:26.049150 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.049132 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-envoy" pod "575ac171-83b1-4d32-bc19-9a039f885c0a" (UID: "575ac171-83b1-4d32-bc19-9a039f885c0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:26.148255 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.148222 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:26.148255 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.148250 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/575ac171-83b1-4d32-bc19-9a039f885c0a-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:26.148255 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.148260 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:26.148511 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.148270 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf577\" (UniqueName: \"kubernetes.io/projected/575ac171-83b1-4d32-bc19-9a039f885c0a-kube-api-access-bf577\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:26.148511 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.148280 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/575ac171-83b1-4d32-bc19-9a039f885c0a-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:26.148511 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.148291 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/575ac171-83b1-4d32-bc19-9a039f885c0a-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:26.176465 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.176430 5601 generic.go:356] "Generic (PLEG): container finished" podID="575ac171-83b1-4d32-bc19-9a039f885c0a" containerID="f71922eb9dcdf9562dce8036c729ff9ba3fc3d85fb32951c45edacf28b8ddb8f" exitCode=0 Nov 02 18:29:26.176612 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.176486 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" Nov 02 18:29:26.176612 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.176503 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" event={"ID":"575ac171-83b1-4d32-bc19-9a039f885c0a","Type":"ContainerDied","Data":"f71922eb9dcdf9562dce8036c729ff9ba3fc3d85fb32951c45edacf28b8ddb8f"} Nov 02 18:29:26.176612 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.176554 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9" event={"ID":"575ac171-83b1-4d32-bc19-9a039f885c0a","Type":"ContainerDied","Data":"7a3d54e04b72d93d6298a7f41cb30e31a97e81d6dc79fda3063867351f937356"} Nov 02 18:29:26.176612 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.176585 5601 scope.go:122] "RemoveContainer" containerID="f71922eb9dcdf9562dce8036c729ff9ba3fc3d85fb32951c45edacf28b8ddb8f" Nov 02 18:29:26.189163 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.189147 5601 scope.go:122] "RemoveContainer" containerID="feeeaf737ec76220a88d7b95c0b56dbf3b06b18d343cfa406fd59269a1adf883" Nov 02 18:29:26.198704 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.198684 5601 scope.go:122] "RemoveContainer" containerID="f71922eb9dcdf9562dce8036c729ff9ba3fc3d85fb32951c45edacf28b8ddb8f" Nov 02 18:29:26.198974 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:29:26.198953 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71922eb9dcdf9562dce8036c729ff9ba3fc3d85fb32951c45edacf28b8ddb8f\": container with ID starting with f71922eb9dcdf9562dce8036c729ff9ba3fc3d85fb32951c45edacf28b8ddb8f not found: ID does not exist" containerID="f71922eb9dcdf9562dce8036c729ff9ba3fc3d85fb32951c45edacf28b8ddb8f" Nov 02 18:29:26.199052 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.198985 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71922eb9dcdf9562dce8036c729ff9ba3fc3d85fb32951c45edacf28b8ddb8f"} err="failed to get container status \"f71922eb9dcdf9562dce8036c729ff9ba3fc3d85fb32951c45edacf28b8ddb8f\": rpc error: code = NotFound desc = could not find container \"f71922eb9dcdf9562dce8036c729ff9ba3fc3d85fb32951c45edacf28b8ddb8f\": container with ID starting with f71922eb9dcdf9562dce8036c729ff9ba3fc3d85fb32951c45edacf28b8ddb8f not found: ID does not exist" Nov 02 18:29:26.199052 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.199008 5601 scope.go:122] "RemoveContainer" containerID="feeeaf737ec76220a88d7b95c0b56dbf3b06b18d343cfa406fd59269a1adf883" Nov 02 18:29:26.199286 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:29:26.199266 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feeeaf737ec76220a88d7b95c0b56dbf3b06b18d343cfa406fd59269a1adf883\": container with ID starting with feeeaf737ec76220a88d7b95c0b56dbf3b06b18d343cfa406fd59269a1adf883 not found: ID does not exist" containerID="feeeaf737ec76220a88d7b95c0b56dbf3b06b18d343cfa406fd59269a1adf883" Nov 02 18:29:26.199412 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.199288 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feeeaf737ec76220a88d7b95c0b56dbf3b06b18d343cfa406fd59269a1adf883"} err="failed to get container status \"feeeaf737ec76220a88d7b95c0b56dbf3b06b18d343cfa406fd59269a1adf883\": rpc error: code = NotFound desc = could not find container \"feeeaf737ec76220a88d7b95c0b56dbf3b06b18d343cfa406fd59269a1adf883\": container with ID starting with feeeaf737ec76220a88d7b95c0b56dbf3b06b18d343cfa406fd59269a1adf883 not found: ID does not exist" Nov 02 18:29:26.202728 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.202702 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9"] Nov 02 18:29:26.205567 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:26.205545 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-327608-openshift-default-569bd4db89-gqhl9"] Nov 02 18:29:27.426251 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:27.426214 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="575ac171-83b1-4d32-bc19-9a039f885c0a" path="/var/lib/kubelet/pods/575ac171-83b1-4d32-bc19-9a039f885c0a/volumes" Nov 02 18:29:28.989733 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:28.989705 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 18:29:29.087810 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.087722 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-credential-socket\") pod \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " Nov 02 18:29:29.087810 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.087766 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istiod-ca-cert\") pod \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " Nov 02 18:29:29.087810 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.087797 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-token\" (UniqueName: \"kubernetes.io/projected/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-token\") pod \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " Nov 02 18:29:29.088088 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.087876 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-podinfo\") pod \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " Nov 02 18:29:29.088088 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.087909 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-envoy\") pod \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " Nov 02 18:29:29.088088 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.088018 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-credential-socket" pod "0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" (UID: "0d7f63eb-dd21-48bf-a0d5-2919ca7e668a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:29.088088 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.088041 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-data\") pod \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " Nov 02 18:29:29.088318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.088091 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-kube-api-access-v9jrp\" (UniqueName: \"kubernetes.io/projected/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-kube-api-access-v9jrp\") pod \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " Nov 02 18:29:29.088318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.088165 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-workload-socket\") pod \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " Nov 02 18:29:29.088318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.088207 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-workload-certs\") pod \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\" (UID: \"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a\") " Nov 02 18:29:29.088318 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.088263 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istiod-ca-cert" pod "0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" (UID: "0d7f63eb-dd21-48bf-a0d5-2919ca7e668a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:29.089019 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.088563 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-data" pod "0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" (UID: "0d7f63eb-dd21-48bf-a0d5-2919ca7e668a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:29.089019 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.088967 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-workload-socket" pod "0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" (UID: "0d7f63eb-dd21-48bf-a0d5-2919ca7e668a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:29.089460 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.089113 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:29.089460 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.089174 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:29.089460 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.089189 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:29.089460 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.089435 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-workload-certs" pod "0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" (UID: "0d7f63eb-dd21-48bf-a0d5-2919ca7e668a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:29.090379 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.090350 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-podinfo" pod "0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" (UID: "0d7f63eb-dd21-48bf-a0d5-2919ca7e668a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:29:29.090576 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.090553 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-envoy" pod "0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" (UID: "0d7f63eb-dd21-48bf-a0d5-2919ca7e668a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:29.090943 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.090913 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-kube-api-access-v9jrp" pod "0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" (UID: "0d7f63eb-dd21-48bf-a0d5-2919ca7e668a"). InnerVolumeSpecName "kube-api-access-v9jrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:29.091157 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.091133 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-token" pod "0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" (UID: "0d7f63eb-dd21-48bf-a0d5-2919ca7e668a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:29.197639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.194682 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:29.197639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.194725 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:29.197639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.194740 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:29.197639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.194754 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:29.197639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.194768 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:29.197639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.194783 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9jrp\" (UniqueName: \"kubernetes.io/projected/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a-kube-api-access-v9jrp\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:29.211563 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.211517 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd"] Nov 02 18:29:29.214608 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.214546 5601 generic.go:356] "Generic (PLEG): container finished" podID="0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" containerID="84c9171fa2cacf21d0e7db8324e5f6ff5a09e7c2938af1e60bb3b39ddacf01d5" exitCode=0 Nov 02 18:29:29.214796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.214752 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" event={"ID":"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a","Type":"ContainerDied","Data":"84c9171fa2cacf21d0e7db8324e5f6ff5a09e7c2938af1e60bb3b39ddacf01d5"} Nov 02 18:29:29.214857 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.214803 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" event={"ID":"0d7f63eb-dd21-48bf-a0d5-2919ca7e668a","Type":"ContainerDied","Data":"66909fbf362cc2b6764667d012e84a9dbd483231d7e9a179c10cc75c89bd9f34"} Nov 02 18:29:29.214857 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.214834 5601 scope.go:122] "RemoveContainer" containerID="84c9171fa2cacf21d0e7db8324e5f6ff5a09e7c2938af1e60bb3b39ddacf01d5" Nov 02 18:29:29.216006 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.215828 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl" Nov 02 18:29:29.216611 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.216589 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm"] Nov 02 18:29:29.231119 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.230985 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc"] Nov 02 18:29:29.232754 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.232732 5601 scope.go:122] "RemoveContainer" containerID="93dda336277be2c29190b693bf337279c96299936d5c7ffcb387bba991a6455e" Nov 02 18:29:29.254525 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.254506 5601 scope.go:122] "RemoveContainer" containerID="84c9171fa2cacf21d0e7db8324e5f6ff5a09e7c2938af1e60bb3b39ddacf01d5" Nov 02 18:29:29.254943 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:29:29.254914 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84c9171fa2cacf21d0e7db8324e5f6ff5a09e7c2938af1e60bb3b39ddacf01d5\": container with ID starting with 84c9171fa2cacf21d0e7db8324e5f6ff5a09e7c2938af1e60bb3b39ddacf01d5 not found: ID does not exist" containerID="84c9171fa2cacf21d0e7db8324e5f6ff5a09e7c2938af1e60bb3b39ddacf01d5" Nov 02 18:29:29.255050 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.254951 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84c9171fa2cacf21d0e7db8324e5f6ff5a09e7c2938af1e60bb3b39ddacf01d5"} err="failed to get container status \"84c9171fa2cacf21d0e7db8324e5f6ff5a09e7c2938af1e60bb3b39ddacf01d5\": rpc error: code = NotFound desc = could not find container \"84c9171fa2cacf21d0e7db8324e5f6ff5a09e7c2938af1e60bb3b39ddacf01d5\": container with ID starting with 84c9171fa2cacf21d0e7db8324e5f6ff5a09e7c2938af1e60bb3b39ddacf01d5 not found: ID does not exist" Nov 02 18:29:29.255050 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.254968 5601 scope.go:122] "RemoveContainer" containerID="93dda336277be2c29190b693bf337279c96299936d5c7ffcb387bba991a6455e" Nov 02 18:29:29.255245 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:29:29.255227 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93dda336277be2c29190b693bf337279c96299936d5c7ffcb387bba991a6455e\": container with ID starting with 93dda336277be2c29190b693bf337279c96299936d5c7ffcb387bba991a6455e not found: ID does not exist" containerID="93dda336277be2c29190b693bf337279c96299936d5c7ffcb387bba991a6455e" Nov 02 18:29:29.255288 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.255251 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93dda336277be2c29190b693bf337279c96299936d5c7ffcb387bba991a6455e"} err="failed to get container status \"93dda336277be2c29190b693bf337279c96299936d5c7ffcb387bba991a6455e\": rpc error: code = NotFound desc = could not find container \"93dda336277be2c29190b693bf337279c96299936d5c7ffcb387bba991a6455e\": container with ID starting with 93dda336277be2c29190b693bf337279c96299936d5c7ffcb387bba991a6455e not found: ID does not exist" Nov 02 18:29:29.266340 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.266317 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl"] Nov 02 18:29:29.271754 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.271702 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2961d5-openshift-default-76d5b87c4f-l75rl"] Nov 02 18:29:29.427119 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:29.427086 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" path="/var/lib/kubelet/pods/0d7f63eb-dd21-48bf-a0d5-2919ca7e668a/volumes" Nov 02 18:29:31.142636 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:31.142589 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2"] Nov 02 18:29:31.143163 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:31.142849 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" podUID="e5f3c081-92f7-4903-a974-779520012039" containerName="maas-api" containerID="cri-o://74859d646310f1000859f014cc10b72a75c05a7a507a07937eade4f6cc9f589d" gracePeriod=30 Nov 02 18:29:31.569539 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:31.569503 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" Nov 02 18:29:31.606385 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:31.602389 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f3c081-92f7-4903-a974-779520012039-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f3c081-92f7-4903-a974-779520012039-openshift-service-ca\") pod \"e5f3c081-92f7-4903-a974-779520012039\" (UID: \"e5f3c081-92f7-4903-a974-779520012039\") " Nov 02 18:29:31.609459 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:31.607147 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f3c081-92f7-4903-a974-779520012039-openshift-service-ca" pod "e5f3c081-92f7-4903-a974-779520012039" (UID: "e5f3c081-92f7-4903-a974-779520012039"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:31.609677 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:31.609566 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f3c081-92f7-4903-a974-779520012039-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f3c081-92f7-4903-a974-779520012039-maas-api-tls\") pod \"e5f3c081-92f7-4903-a974-779520012039\" (UID: \"e5f3c081-92f7-4903-a974-779520012039\") " Nov 02 18:29:31.610159 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:31.610131 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f3c081-92f7-4903-a974-779520012039-kube-api-access-z6gsq\" (UniqueName: \"kubernetes.io/projected/e5f3c081-92f7-4903-a974-779520012039-kube-api-access-z6gsq\") pod \"e5f3c081-92f7-4903-a974-779520012039\" (UID: \"e5f3c081-92f7-4903-a974-779520012039\") " Nov 02 18:29:31.614723 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:31.610596 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f3c081-92f7-4903-a974-779520012039-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f3c081-92f7-4903-a974-779520012039-metrics-certs\") pod \"e5f3c081-92f7-4903-a974-779520012039\" (UID: \"e5f3c081-92f7-4903-a974-779520012039\") " Nov 02 18:29:31.614723 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:31.611190 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f3c081-92f7-4903-a974-779520012039-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:31.614928 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:31.614734 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f3c081-92f7-4903-a974-779520012039-metrics-certs" pod "e5f3c081-92f7-4903-a974-779520012039" (UID: "e5f3c081-92f7-4903-a974-779520012039"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:31.633974 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:31.633910 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f3c081-92f7-4903-a974-779520012039-maas-api-tls" pod "e5f3c081-92f7-4903-a974-779520012039" (UID: "e5f3c081-92f7-4903-a974-779520012039"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:31.634733 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:31.634697 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f3c081-92f7-4903-a974-779520012039-kube-api-access-z6gsq" pod "e5f3c081-92f7-4903-a974-779520012039" (UID: "e5f3c081-92f7-4903-a974-779520012039"). InnerVolumeSpecName "kube-api-access-z6gsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:31.713350 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:31.713309 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6gsq\" (UniqueName: \"kubernetes.io/projected/e5f3c081-92f7-4903-a974-779520012039-kube-api-access-z6gsq\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:31.713350 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:31.713350 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f3c081-92f7-4903-a974-779520012039-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:31.713573 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:31.713365 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f3c081-92f7-4903-a974-779520012039-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:32.243101 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:32.243059 5601 generic.go:356] "Generic (PLEG): container finished" podID="e5f3c081-92f7-4903-a974-779520012039" containerID="74859d646310f1000859f014cc10b72a75c05a7a507a07937eade4f6cc9f589d" exitCode=0 Nov 02 18:29:32.243563 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:32.243117 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" Nov 02 18:29:32.243563 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:32.243144 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" event={"ID":"e5f3c081-92f7-4903-a974-779520012039","Type":"ContainerDied","Data":"74859d646310f1000859f014cc10b72a75c05a7a507a07937eade4f6cc9f589d"} Nov 02 18:29:32.243563 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:32.243203 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2" event={"ID":"e5f3c081-92f7-4903-a974-779520012039","Type":"ContainerDied","Data":"36e94d28ab9e1835fa261e870987167ca3602aece640c430854941d065f7ed90"} Nov 02 18:29:32.243563 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:32.243233 5601 scope.go:122] "RemoveContainer" containerID="74859d646310f1000859f014cc10b72a75c05a7a507a07937eade4f6cc9f589d" Nov 02 18:29:32.255328 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:32.255310 5601 scope.go:122] "RemoveContainer" containerID="74859d646310f1000859f014cc10b72a75c05a7a507a07937eade4f6cc9f589d" Nov 02 18:29:32.255608 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:29:32.255583 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74859d646310f1000859f014cc10b72a75c05a7a507a07937eade4f6cc9f589d\": container with ID starting with 74859d646310f1000859f014cc10b72a75c05a7a507a07937eade4f6cc9f589d not found: ID does not exist" containerID="74859d646310f1000859f014cc10b72a75c05a7a507a07937eade4f6cc9f589d" Nov 02 18:29:32.255685 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:32.255635 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74859d646310f1000859f014cc10b72a75c05a7a507a07937eade4f6cc9f589d"} err="failed to get container status \"74859d646310f1000859f014cc10b72a75c05a7a507a07937eade4f6cc9f589d\": rpc error: code = NotFound desc = could not find container \"74859d646310f1000859f014cc10b72a75c05a7a507a07937eade4f6cc9f589d\": container with ID starting with 74859d646310f1000859f014cc10b72a75c05a7a507a07937eade4f6cc9f589d not found: ID does not exist" Nov 02 18:29:32.267102 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:32.267078 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2"] Nov 02 18:29:32.271540 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:32.271518 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0f0baf-76c957784b-wqjv2"] Nov 02 18:29:33.427672 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:33.427609 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f3c081-92f7-4903-a974-779520012039" path="/var/lib/kubelet/pods/e5f3c081-92f7-4903-a974-779520012039/volumes" Nov 02 18:29:33.477366 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:33.477326 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf"] Nov 02 18:29:33.477873 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:33.477843 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" podUID="519549a8-7b5f-46f6-9e58-428c162bbe63" containerName="maas-api" containerID="cri-o://1e8d78114f538231e03a451cc17ef30dc9a6c751ff055a2542a78f171c4d0717" gracePeriod=30 Nov 02 18:29:33.532401 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:33.532358 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" podUID="519549a8-7b5f-46f6-9e58-428c162bbe63" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.132:8443/health\": dial tcp 10.134.0.132:8443: connect: connection refused" Nov 02 18:29:33.900042 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:33.899998 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" Nov 02 18:29:34.072155 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.072123 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/519549a8-7b5f-46f6-9e58-428c162bbe63-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519549a8-7b5f-46f6-9e58-428c162bbe63-openshift-service-ca\") pod \"519549a8-7b5f-46f6-9e58-428c162bbe63\" (UID: \"519549a8-7b5f-46f6-9e58-428c162bbe63\") " Nov 02 18:29:34.072345 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.072177 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519549a8-7b5f-46f6-9e58-428c162bbe63-kube-api-access-dplzv\" (UniqueName: \"kubernetes.io/projected/519549a8-7b5f-46f6-9e58-428c162bbe63-kube-api-access-dplzv\") pod \"519549a8-7b5f-46f6-9e58-428c162bbe63\" (UID: \"519549a8-7b5f-46f6-9e58-428c162bbe63\") " Nov 02 18:29:34.072345 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.072313 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519549a8-7b5f-46f6-9e58-428c162bbe63-metrics-certs\" (UniqueName: \"kubernetes.io/secret/519549a8-7b5f-46f6-9e58-428c162bbe63-metrics-certs\") pod \"519549a8-7b5f-46f6-9e58-428c162bbe63\" (UID: \"519549a8-7b5f-46f6-9e58-428c162bbe63\") " Nov 02 18:29:34.072668 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.072494 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519549a8-7b5f-46f6-9e58-428c162bbe63-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519549a8-7b5f-46f6-9e58-428c162bbe63-maas-api-tls\") pod \"519549a8-7b5f-46f6-9e58-428c162bbe63\" (UID: \"519549a8-7b5f-46f6-9e58-428c162bbe63\") " Nov 02 18:29:34.072790 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.072662 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519549a8-7b5f-46f6-9e58-428c162bbe63-openshift-service-ca" pod "519549a8-7b5f-46f6-9e58-428c162bbe63" (UID: "519549a8-7b5f-46f6-9e58-428c162bbe63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:34.073746 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.073713 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519549a8-7b5f-46f6-9e58-428c162bbe63-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:34.074934 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.074910 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519549a8-7b5f-46f6-9e58-428c162bbe63-kube-api-access-dplzv" pod "519549a8-7b5f-46f6-9e58-428c162bbe63" (UID: "519549a8-7b5f-46f6-9e58-428c162bbe63"). InnerVolumeSpecName "kube-api-access-dplzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:34.075053 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.075008 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519549a8-7b5f-46f6-9e58-428c162bbe63-metrics-certs" pod "519549a8-7b5f-46f6-9e58-428c162bbe63" (UID: "519549a8-7b5f-46f6-9e58-428c162bbe63"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:34.075217 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.075191 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519549a8-7b5f-46f6-9e58-428c162bbe63-maas-api-tls" pod "519549a8-7b5f-46f6-9e58-428c162bbe63" (UID: "519549a8-7b5f-46f6-9e58-428c162bbe63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:34.175246 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.175217 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519549a8-7b5f-46f6-9e58-428c162bbe63-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:34.175246 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.175247 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dplzv\" (UniqueName: \"kubernetes.io/projected/519549a8-7b5f-46f6-9e58-428c162bbe63-kube-api-access-dplzv\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:34.175435 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.175257 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519549a8-7b5f-46f6-9e58-428c162bbe63-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:34.260527 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.260489 5601 generic.go:356] "Generic (PLEG): container finished" podID="519549a8-7b5f-46f6-9e58-428c162bbe63" containerID="1e8d78114f538231e03a451cc17ef30dc9a6c751ff055a2542a78f171c4d0717" exitCode=0 Nov 02 18:29:34.260739 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.260543 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" Nov 02 18:29:34.260739 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.260572 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" event={"ID":"519549a8-7b5f-46f6-9e58-428c162bbe63","Type":"ContainerDied","Data":"1e8d78114f538231e03a451cc17ef30dc9a6c751ff055a2542a78f171c4d0717"} Nov 02 18:29:34.260739 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.260644 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf" event={"ID":"519549a8-7b5f-46f6-9e58-428c162bbe63","Type":"ContainerDied","Data":"75d6736a67196d4bb740a33585b0ec455287fe5999744b9f5492735b8d2bdf05"} Nov 02 18:29:34.260739 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.260675 5601 scope.go:122] "RemoveContainer" containerID="1e8d78114f538231e03a451cc17ef30dc9a6c751ff055a2542a78f171c4d0717" Nov 02 18:29:34.273319 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.273292 5601 scope.go:122] "RemoveContainer" containerID="1e8d78114f538231e03a451cc17ef30dc9a6c751ff055a2542a78f171c4d0717" Nov 02 18:29:34.273665 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:29:34.273605 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8d78114f538231e03a451cc17ef30dc9a6c751ff055a2542a78f171c4d0717\": container with ID starting with 1e8d78114f538231e03a451cc17ef30dc9a6c751ff055a2542a78f171c4d0717 not found: ID does not exist" containerID="1e8d78114f538231e03a451cc17ef30dc9a6c751ff055a2542a78f171c4d0717" Nov 02 18:29:34.273755 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.273675 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8d78114f538231e03a451cc17ef30dc9a6c751ff055a2542a78f171c4d0717"} err="failed to get container status \"1e8d78114f538231e03a451cc17ef30dc9a6c751ff055a2542a78f171c4d0717\": rpc error: code = NotFound desc = could not find container \"1e8d78114f538231e03a451cc17ef30dc9a6c751ff055a2542a78f171c4d0717\": container with ID starting with 1e8d78114f538231e03a451cc17ef30dc9a6c751ff055a2542a78f171c4d0717 not found: ID does not exist" Nov 02 18:29:34.290826 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.290798 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf"] Nov 02 18:29:34.293810 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:34.293788 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8e574-66c69cffc8-sg4cf"] Nov 02 18:29:34.485696 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:29:34.485601 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:29:35.470641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:35.442087 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="519549a8-7b5f-46f6-9e58-428c162bbe63" path="/var/lib/kubelet/pods/519549a8-7b5f-46f6-9e58-428c162bbe63/volumes" Nov 02 18:29:44.271691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.259481 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" containerID="cri-o://ebadcc1635e2bd3139ffe8d9b2ec38c00de6651b8a862d7fc92e5ebf0a9bed96" gracePeriod=45 Nov 02 18:29:44.271691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.268800 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" containerID="cri-o://26e80b94447307bbb7d9c68eff1e3adc0eaf57963b87b71a0dea35da0f08abf2" gracePeriod=45 Nov 02 18:29:44.290005 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.289184 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" containerID="cri-o://7f25e694f509f0931bf2bb9ff2aadd979adec71353a14975b62176efb4e320bf" gracePeriod=45 Nov 02 18:29:44.311304 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.310921 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": dial tcp 10.134.0.73:8000: connect: connection refused" Nov 02 18:29:44.324361 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.324319 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": dial tcp 10.134.0.58:8000: connect: connection refused" Nov 02 18:29:44.633944 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.633551 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 18:29:44.638368 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.638341 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 18:29:44.644706 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.641648 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 18:29:44.746410 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.746369 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-model-cache\") pod \"436816c2-2c00-4cad-aac7-298db2f5e8d5\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " Nov 02 18:29:44.746608 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.746446 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d064f41-fece-402e-a07f-8515424b2c79-kube-api-access-nmv84\" (UniqueName: \"kubernetes.io/projected/5d064f41-fece-402e-a07f-8515424b2c79-kube-api-access-nmv84\") pod \"5d064f41-fece-402e-a07f-8515424b2c79\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " Nov 02 18:29:44.746608 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.746548 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-dshm\") pod \"3b3e335e-4328-429a-9433-728436384eba\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " Nov 02 18:29:44.746779 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.746607 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-model-cache\") pod \"5d064f41-fece-402e-a07f-8515424b2c79\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " Nov 02 18:29:44.746779 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.746666 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-dshm\") pod \"5d064f41-fece-402e-a07f-8515424b2c79\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " Nov 02 18:29:44.746779 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.746706 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-tmp-dir\") pod \"436816c2-2c00-4cad-aac7-298db2f5e8d5\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " Nov 02 18:29:44.746779 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.746733 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-model-cache\") pod \"3b3e335e-4328-429a-9433-728436384eba\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " Nov 02 18:29:44.746779 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.746772 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b3e335e-4328-429a-9433-728436384eba-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3e335e-4328-429a-9433-728436384eba-tls-certs\") pod \"3b3e335e-4328-429a-9433-728436384eba\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " Nov 02 18:29:44.747048 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.746819 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b3e335e-4328-429a-9433-728436384eba-kube-api-access-vxqwb\" (UniqueName: \"kubernetes.io/projected/3b3e335e-4328-429a-9433-728436384eba-kube-api-access-vxqwb\") pod \"3b3e335e-4328-429a-9433-728436384eba\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " Nov 02 18:29:44.747048 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.746863 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-tmp-dir\") pod \"5d064f41-fece-402e-a07f-8515424b2c79\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " Nov 02 18:29:44.747048 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.746897 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-home\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-home\") pod \"5d064f41-fece-402e-a07f-8515424b2c79\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " Nov 02 18:29:44.747048 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.746945 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436816c2-2c00-4cad-aac7-298db2f5e8d5-kube-api-access-cjzt5\" (UniqueName: \"kubernetes.io/projected/436816c2-2c00-4cad-aac7-298db2f5e8d5-kube-api-access-cjzt5\") pod \"436816c2-2c00-4cad-aac7-298db2f5e8d5\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " Nov 02 18:29:44.747048 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.746979 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-tmp-dir\") pod \"3b3e335e-4328-429a-9433-728436384eba\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " Nov 02 18:29:44.747048 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.747007 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-dshm\") pod \"436816c2-2c00-4cad-aac7-298db2f5e8d5\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " Nov 02 18:29:44.747487 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.747135 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d064f41-fece-402e-a07f-8515424b2c79-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d064f41-fece-402e-a07f-8515424b2c79-tls-certs\") pod \"5d064f41-fece-402e-a07f-8515424b2c79\" (UID: \"5d064f41-fece-402e-a07f-8515424b2c79\") " Nov 02 18:29:44.747487 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.747191 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-home\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-home\") pod \"436816c2-2c00-4cad-aac7-298db2f5e8d5\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " Nov 02 18:29:44.747487 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.747221 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/436816c2-2c00-4cad-aac7-298db2f5e8d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/436816c2-2c00-4cad-aac7-298db2f5e8d5-tls-certs\") pod \"436816c2-2c00-4cad-aac7-298db2f5e8d5\" (UID: \"436816c2-2c00-4cad-aac7-298db2f5e8d5\") " Nov 02 18:29:44.747487 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.747289 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-home\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-home\") pod \"3b3e335e-4328-429a-9433-728436384eba\" (UID: \"3b3e335e-4328-429a-9433-728436384eba\") " Nov 02 18:29:44.748369 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.748338 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-home" pod "3b3e335e-4328-429a-9433-728436384eba" (UID: "3b3e335e-4328-429a-9433-728436384eba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:44.748587 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.748565 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-tmp-dir" pod "5d064f41-fece-402e-a07f-8515424b2c79" (UID: "5d064f41-fece-402e-a07f-8515424b2c79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:44.748829 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.748807 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-home" pod "5d064f41-fece-402e-a07f-8515424b2c79" (UID: "5d064f41-fece-402e-a07f-8515424b2c79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:44.749202 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.749174 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-tmp-dir" pod "436816c2-2c00-4cad-aac7-298db2f5e8d5" (UID: "436816c2-2c00-4cad-aac7-298db2f5e8d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:44.749492 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.749466 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-model-cache" pod "3b3e335e-4328-429a-9433-728436384eba" (UID: "3b3e335e-4328-429a-9433-728436384eba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:44.749820 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.749797 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-model-cache" pod "436816c2-2c00-4cad-aac7-298db2f5e8d5" (UID: "436816c2-2c00-4cad-aac7-298db2f5e8d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:44.750325 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.750300 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-model-cache" pod "5d064f41-fece-402e-a07f-8515424b2c79" (UID: "5d064f41-fece-402e-a07f-8515424b2c79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:44.751552 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.751414 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d064f41-fece-402e-a07f-8515424b2c79-kube-api-access-nmv84" pod "5d064f41-fece-402e-a07f-8515424b2c79" (UID: "5d064f41-fece-402e-a07f-8515424b2c79"). InnerVolumeSpecName "kube-api-access-nmv84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:44.752262 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.751997 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-home" pod "436816c2-2c00-4cad-aac7-298db2f5e8d5" (UID: "436816c2-2c00-4cad-aac7-298db2f5e8d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:44.754668 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.754509 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/436816c2-2c00-4cad-aac7-298db2f5e8d5-tls-certs" pod "436816c2-2c00-4cad-aac7-298db2f5e8d5" (UID: "436816c2-2c00-4cad-aac7-298db2f5e8d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:44.755916 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.755026 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-dshm" pod "5d064f41-fece-402e-a07f-8515424b2c79" (UID: "5d064f41-fece-402e-a07f-8515424b2c79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:44.755916 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.755862 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-dshm" pod "436816c2-2c00-4cad-aac7-298db2f5e8d5" (UID: "436816c2-2c00-4cad-aac7-298db2f5e8d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:44.756175 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.756122 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-tmp-dir" pod "3b3e335e-4328-429a-9433-728436384eba" (UID: "3b3e335e-4328-429a-9433-728436384eba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:44.756367 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.756266 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3e335e-4328-429a-9433-728436384eba-kube-api-access-vxqwb" pod "3b3e335e-4328-429a-9433-728436384eba" (UID: "3b3e335e-4328-429a-9433-728436384eba"). InnerVolumeSpecName "kube-api-access-vxqwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:44.756586 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.756562 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-dshm" pod "3b3e335e-4328-429a-9433-728436384eba" (UID: "3b3e335e-4328-429a-9433-728436384eba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:44.756914 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.756881 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d064f41-fece-402e-a07f-8515424b2c79-tls-certs" pod "5d064f41-fece-402e-a07f-8515424b2c79" (UID: "5d064f41-fece-402e-a07f-8515424b2c79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:44.756914 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.756889 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b3e335e-4328-429a-9433-728436384eba-tls-certs" pod "3b3e335e-4328-429a-9433-728436384eba" (UID: "3b3e335e-4328-429a-9433-728436384eba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:44.758430 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.758405 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436816c2-2c00-4cad-aac7-298db2f5e8d5-kube-api-access-cjzt5" pod "436816c2-2c00-4cad-aac7-298db2f5e8d5" (UID: "436816c2-2c00-4cad-aac7-298db2f5e8d5"). InnerVolumeSpecName "kube-api-access-cjzt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:44.850301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.849966 5601 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-dshm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:44.850301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.850002 5601 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d064f41-fece-402e-a07f-8515424b2c79-tls-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:44.850301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.850016 5601 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-home\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:44.850301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.850030 5601 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436816c2-2c00-4cad-aac7-298db2f5e8d5-tls-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:44.850301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.850042 5601 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-home\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:44.850301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.850059 5601 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-model-cache\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:44.850301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.850075 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmv84\" (UniqueName: \"kubernetes.io/projected/5d064f41-fece-402e-a07f-8515424b2c79-kube-api-access-nmv84\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:44.850301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.850091 5601 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-dshm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:44.850301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.850104 5601 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-model-cache\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:44.850301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.850116 5601 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-dshm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:44.850301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.850129 5601 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436816c2-2c00-4cad-aac7-298db2f5e8d5-tmp-dir\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:44.850301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.850144 5601 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-model-cache\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:44.850301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.850158 5601 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3e335e-4328-429a-9433-728436384eba-tls-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:44.850301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.850171 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxqwb\" (UniqueName: \"kubernetes.io/projected/3b3e335e-4328-429a-9433-728436384eba-kube-api-access-vxqwb\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:44.850301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.850185 5601 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-tmp-dir\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:44.850301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.850198 5601 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d064f41-fece-402e-a07f-8515424b2c79-home\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:44.850301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.850211 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjzt5\" (UniqueName: \"kubernetes.io/projected/436816c2-2c00-4cad-aac7-298db2f5e8d5-kube-api-access-cjzt5\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:44.850301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:44.850224 5601 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3e335e-4328-429a-9433-728436384eba-tmp-dir\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:45.372337 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.372290 5601 generic.go:356] "Generic (PLEG): container finished" podID="3b3e335e-4328-429a-9433-728436384eba" containerID="7f25e694f509f0931bf2bb9ff2aadd979adec71353a14975b62176efb4e320bf" exitCode=0 Nov 02 18:29:45.372830 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.372342 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" Nov 02 18:29:45.372830 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.372351 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" event={"ID":"3b3e335e-4328-429a-9433-728436384eba","Type":"ContainerDied","Data":"7f25e694f509f0931bf2bb9ff2aadd979adec71353a14975b62176efb4e320bf"} Nov 02 18:29:45.372830 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.372409 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm" event={"ID":"3b3e335e-4328-429a-9433-728436384eba","Type":"ContainerDied","Data":"9a78dcc22ed221b448cf21e8678b5f1fba2a3882d83e2f0f6d3d42b7ee2d1ea3"} Nov 02 18:29:45.372830 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.372436 5601 scope.go:122] "RemoveContainer" containerID="7f25e694f509f0931bf2bb9ff2aadd979adec71353a14975b62176efb4e320bf" Nov 02 18:29:45.374433 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.374412 5601 generic.go:356] "Generic (PLEG): container finished" podID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerID="26e80b94447307bbb7d9c68eff1e3adc0eaf57963b87b71a0dea35da0f08abf2" exitCode=0 Nov 02 18:29:45.374595 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.374576 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" event={"ID":"436816c2-2c00-4cad-aac7-298db2f5e8d5","Type":"ContainerDied","Data":"26e80b94447307bbb7d9c68eff1e3adc0eaf57963b87b71a0dea35da0f08abf2"} Nov 02 18:29:45.374595 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.374587 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" Nov 02 18:29:45.374789 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.374610 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc" event={"ID":"436816c2-2c00-4cad-aac7-298db2f5e8d5","Type":"ContainerDied","Data":"335a70147f49963416111982d947dea6b6157741f0bcc25730d29f7cc532ead3"} Nov 02 18:29:45.376730 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.376707 5601 generic.go:356] "Generic (PLEG): container finished" podID="5d064f41-fece-402e-a07f-8515424b2c79" containerID="ebadcc1635e2bd3139ffe8d9b2ec38c00de6651b8a862d7fc92e5ebf0a9bed96" exitCode=0 Nov 02 18:29:45.376846 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.376741 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" event={"ID":"5d064f41-fece-402e-a07f-8515424b2c79","Type":"ContainerDied","Data":"ebadcc1635e2bd3139ffe8d9b2ec38c00de6651b8a862d7fc92e5ebf0a9bed96"} Nov 02 18:29:45.376846 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.376768 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" event={"ID":"5d064f41-fece-402e-a07f-8515424b2c79","Type":"ContainerDied","Data":"cc1a453576a5fde1729406584f452fbe66a04e2f16db4c4eb504463a7cc9415d"} Nov 02 18:29:45.376846 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.376824 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd" Nov 02 18:29:45.389630 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.387281 5601 scope.go:122] "RemoveContainer" containerID="7f25e694f509f0931bf2bb9ff2aadd979adec71353a14975b62176efb4e320bf" Nov 02 18:29:45.389630 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:29:45.388265 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f25e694f509f0931bf2bb9ff2aadd979adec71353a14975b62176efb4e320bf\": container with ID starting with 7f25e694f509f0931bf2bb9ff2aadd979adec71353a14975b62176efb4e320bf not found: ID does not exist" containerID="7f25e694f509f0931bf2bb9ff2aadd979adec71353a14975b62176efb4e320bf" Nov 02 18:29:45.389630 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.388296 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f25e694f509f0931bf2bb9ff2aadd979adec71353a14975b62176efb4e320bf"} err="failed to get container status \"7f25e694f509f0931bf2bb9ff2aadd979adec71353a14975b62176efb4e320bf\": rpc error: code = NotFound desc = could not find container \"7f25e694f509f0931bf2bb9ff2aadd979adec71353a14975b62176efb4e320bf\": container with ID starting with 7f25e694f509f0931bf2bb9ff2aadd979adec71353a14975b62176efb4e320bf not found: ID does not exist" Nov 02 18:29:45.389630 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.388316 5601 scope.go:122] "RemoveContainer" containerID="26e80b94447307bbb7d9c68eff1e3adc0eaf57963b87b71a0dea35da0f08abf2" Nov 02 18:29:45.399839 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.399821 5601 scope.go:122] "RemoveContainer" containerID="26e80b94447307bbb7d9c68eff1e3adc0eaf57963b87b71a0dea35da0f08abf2" Nov 02 18:29:45.400110 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:29:45.400090 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e80b94447307bbb7d9c68eff1e3adc0eaf57963b87b71a0dea35da0f08abf2\": container with ID starting with 26e80b94447307bbb7d9c68eff1e3adc0eaf57963b87b71a0dea35da0f08abf2 not found: ID does not exist" containerID="26e80b94447307bbb7d9c68eff1e3adc0eaf57963b87b71a0dea35da0f08abf2" Nov 02 18:29:45.400199 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.400120 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e80b94447307bbb7d9c68eff1e3adc0eaf57963b87b71a0dea35da0f08abf2"} err="failed to get container status \"26e80b94447307bbb7d9c68eff1e3adc0eaf57963b87b71a0dea35da0f08abf2\": rpc error: code = NotFound desc = could not find container \"26e80b94447307bbb7d9c68eff1e3adc0eaf57963b87b71a0dea35da0f08abf2\": container with ID starting with 26e80b94447307bbb7d9c68eff1e3adc0eaf57963b87b71a0dea35da0f08abf2 not found: ID does not exist" Nov 02 18:29:45.400199 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.400145 5601 scope.go:122] "RemoveContainer" containerID="ebadcc1635e2bd3139ffe8d9b2ec38c00de6651b8a862d7fc92e5ebf0a9bed96" Nov 02 18:29:45.403987 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.403964 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm"] Nov 02 18:29:45.406715 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.406694 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/distinct-w1-24cfb6-kserve-797d5fb8cf-sbtxm"] Nov 02 18:29:45.409592 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.409565 5601 scope.go:122] "RemoveContainer" containerID="ebadcc1635e2bd3139ffe8d9b2ec38c00de6651b8a862d7fc92e5ebf0a9bed96" Nov 02 18:29:45.409890 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:29:45.409872 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebadcc1635e2bd3139ffe8d9b2ec38c00de6651b8a862d7fc92e5ebf0a9bed96\": container with ID starting with ebadcc1635e2bd3139ffe8d9b2ec38c00de6651b8a862d7fc92e5ebf0a9bed96 not found: ID does not exist" containerID="ebadcc1635e2bd3139ffe8d9b2ec38c00de6651b8a862d7fc92e5ebf0a9bed96" Nov 02 18:29:45.409987 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.409897 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebadcc1635e2bd3139ffe8d9b2ec38c00de6651b8a862d7fc92e5ebf0a9bed96"} err="failed to get container status \"ebadcc1635e2bd3139ffe8d9b2ec38c00de6651b8a862d7fc92e5ebf0a9bed96\": rpc error: code = NotFound desc = could not find container \"ebadcc1635e2bd3139ffe8d9b2ec38c00de6651b8a862d7fc92e5ebf0a9bed96\": container with ID starting with ebadcc1635e2bd3139ffe8d9b2ec38c00de6651b8a862d7fc92e5ebf0a9bed96 not found: ID does not exist" Nov 02 18:29:45.415841 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.415812 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd"] Nov 02 18:29:45.418385 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.418361 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/distinct-2-w1-24cfb6-kserve-5978fcbbfb-ptbhd"] Nov 02 18:29:45.426730 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.426708 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b3e335e-4328-429a-9433-728436384eba" path="/var/lib/kubelet/pods/3b3e335e-4328-429a-9433-728436384eba/volumes" Nov 02 18:29:45.427179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.427164 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d064f41-fece-402e-a07f-8515424b2c79" path="/var/lib/kubelet/pods/5d064f41-fece-402e-a07f-8515424b2c79/volumes" Nov 02 18:29:45.429061 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.429040 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc"] Nov 02 18:29:45.430719 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:45.430701 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/simulator-w1-24cfb6-kserve-695c745d89-q4nfc"] Nov 02 18:29:47.426240 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:47.426202 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" path="/var/lib/kubelet/pods/436816c2-2c00-4cad-aac7-298db2f5e8d5/volumes" Nov 02 18:29:53.559419 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:53.559378 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj"] Nov 02 18:29:53.559966 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:53.559651 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" podUID="e6e6a7d0-45c4-4fce-aad6-edf7d92c2795" containerName="maas-api" containerID="cri-o://08d24af05b9a7153a6245b16f456439eaa533589531027780d8e50e380ae8206" gracePeriod=30 Nov 02 18:29:54.033268 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.033242 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" Nov 02 18:29:54.123061 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.123028 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-maas-api-tls\") pod \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\" (UID: \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\") " Nov 02 18:29:54.123265 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.123117 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-openshift-service-ca\") pod \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\" (UID: \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\") " Nov 02 18:29:54.123352 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.123293 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-kube-api-access-cw8w7\" (UniqueName: \"kubernetes.io/projected/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-kube-api-access-cw8w7\") pod \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\" (UID: \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\") " Nov 02 18:29:54.123409 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.123387 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-metrics-certs\") pod \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\" (UID: \"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795\") " Nov 02 18:29:54.123468 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.123447 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-openshift-service-ca" pod "e6e6a7d0-45c4-4fce-aad6-edf7d92c2795" (UID: "e6e6a7d0-45c4-4fce-aad6-edf7d92c2795"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:54.123815 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.123796 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:54.125270 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.125241 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-maas-api-tls" pod "e6e6a7d0-45c4-4fce-aad6-edf7d92c2795" (UID: "e6e6a7d0-45c4-4fce-aad6-edf7d92c2795"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:54.125381 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.125366 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-metrics-certs" pod "e6e6a7d0-45c4-4fce-aad6-edf7d92c2795" (UID: "e6e6a7d0-45c4-4fce-aad6-edf7d92c2795"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:54.125428 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.125404 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-kube-api-access-cw8w7" pod "e6e6a7d0-45c4-4fce-aad6-edf7d92c2795" (UID: "e6e6a7d0-45c4-4fce-aad6-edf7d92c2795"). InnerVolumeSpecName "kube-api-access-cw8w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:54.224578 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.224539 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw8w7\" (UniqueName: \"kubernetes.io/projected/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-kube-api-access-cw8w7\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:54.224578 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.224573 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:54.224578 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.224585 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:54.441716 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.441675 5601 generic.go:356] "Generic (PLEG): container finished" podID="e6e6a7d0-45c4-4fce-aad6-edf7d92c2795" containerID="08d24af05b9a7153a6245b16f456439eaa533589531027780d8e50e380ae8206" exitCode=0 Nov 02 18:29:54.441879 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.441727 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" Nov 02 18:29:54.441879 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.441759 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" event={"ID":"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795","Type":"ContainerDied","Data":"08d24af05b9a7153a6245b16f456439eaa533589531027780d8e50e380ae8206"} Nov 02 18:29:54.441879 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.441810 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj" event={"ID":"e6e6a7d0-45c4-4fce-aad6-edf7d92c2795","Type":"ContainerDied","Data":"7c212f720c99b545759f7a57bea0268097a1980a7a40d2d34b5490ef1af6858b"} Nov 02 18:29:54.441879 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.441832 5601 scope.go:122] "RemoveContainer" containerID="08d24af05b9a7153a6245b16f456439eaa533589531027780d8e50e380ae8206" Nov 02 18:29:54.452908 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.452891 5601 scope.go:122] "RemoveContainer" containerID="08d24af05b9a7153a6245b16f456439eaa533589531027780d8e50e380ae8206" Nov 02 18:29:54.453208 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:29:54.453184 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d24af05b9a7153a6245b16f456439eaa533589531027780d8e50e380ae8206\": container with ID starting with 08d24af05b9a7153a6245b16f456439eaa533589531027780d8e50e380ae8206 not found: ID does not exist" containerID="08d24af05b9a7153a6245b16f456439eaa533589531027780d8e50e380ae8206" Nov 02 18:29:54.453310 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.453215 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d24af05b9a7153a6245b16f456439eaa533589531027780d8e50e380ae8206"} err="failed to get container status \"08d24af05b9a7153a6245b16f456439eaa533589531027780d8e50e380ae8206\": rpc error: code = NotFound desc = could not find container \"08d24af05b9a7153a6245b16f456439eaa533589531027780d8e50e380ae8206\": container with ID starting with 08d24af05b9a7153a6245b16f456439eaa533589531027780d8e50e380ae8206 not found: ID does not exist" Nov 02 18:29:54.467509 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.467488 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj"] Nov 02 18:29:54.472661 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:54.472636 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c3c1f-6dcbdd996-kzjtj"] Nov 02 18:29:55.217899 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:55.217863 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs"] Nov 02 18:29:55.219064 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:55.218703 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" podUID="44a346b2-4ddf-475b-9729-6e15d825198d" containerName="maas-api" containerID="cri-o://c89b85385248d06b13208691de04eacfef114e3ac310a94cfa2f76a655755d55" gracePeriod=30 Nov 02 18:29:55.502822 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:55.480173 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6e6a7d0-45c4-4fce-aad6-edf7d92c2795" path="/var/lib/kubelet/pods/e6e6a7d0-45c4-4fce-aad6-edf7d92c2795/volumes" Nov 02 18:29:55.576383 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:55.576344 5601 generic.go:356] "Generic (PLEG): container finished" podID="44a346b2-4ddf-475b-9729-6e15d825198d" containerID="c89b85385248d06b13208691de04eacfef114e3ac310a94cfa2f76a655755d55" exitCode=0 Nov 02 18:29:55.576704 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:55.576668 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" event={"ID":"44a346b2-4ddf-475b-9729-6e15d825198d","Type":"ContainerDied","Data":"c89b85385248d06b13208691de04eacfef114e3ac310a94cfa2f76a655755d55"} Nov 02 18:29:55.733586 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:55.733558 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" Nov 02 18:29:55.788441 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:55.788359 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44a346b2-4ddf-475b-9729-6e15d825198d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44a346b2-4ddf-475b-9729-6e15d825198d-maas-api-tls\") pod \"44a346b2-4ddf-475b-9729-6e15d825198d\" (UID: \"44a346b2-4ddf-475b-9729-6e15d825198d\") " Nov 02 18:29:55.789032 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:55.789006 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44a346b2-4ddf-475b-9729-6e15d825198d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44a346b2-4ddf-475b-9729-6e15d825198d-openshift-service-ca\") pod \"44a346b2-4ddf-475b-9729-6e15d825198d\" (UID: \"44a346b2-4ddf-475b-9729-6e15d825198d\") " Nov 02 18:29:55.789159 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:55.789044 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a346b2-4ddf-475b-9729-6e15d825198d-kube-api-access-9gt8m\" (UniqueName: \"kubernetes.io/projected/44a346b2-4ddf-475b-9729-6e15d825198d-kube-api-access-9gt8m\") pod \"44a346b2-4ddf-475b-9729-6e15d825198d\" (UID: \"44a346b2-4ddf-475b-9729-6e15d825198d\") " Nov 02 18:29:55.789223 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:55.789157 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44a346b2-4ddf-475b-9729-6e15d825198d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a346b2-4ddf-475b-9729-6e15d825198d-metrics-certs\") pod \"44a346b2-4ddf-475b-9729-6e15d825198d\" (UID: \"44a346b2-4ddf-475b-9729-6e15d825198d\") " Nov 02 18:29:55.790239 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:55.790209 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a346b2-4ddf-475b-9729-6e15d825198d-openshift-service-ca" pod "44a346b2-4ddf-475b-9729-6e15d825198d" (UID: "44a346b2-4ddf-475b-9729-6e15d825198d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:55.798292 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:55.798241 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a346b2-4ddf-475b-9729-6e15d825198d-kube-api-access-9gt8m" pod "44a346b2-4ddf-475b-9729-6e15d825198d" (UID: "44a346b2-4ddf-475b-9729-6e15d825198d"). InnerVolumeSpecName "kube-api-access-9gt8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:55.803027 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:55.802977 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44a346b2-4ddf-475b-9729-6e15d825198d-maas-api-tls" pod "44a346b2-4ddf-475b-9729-6e15d825198d" (UID: "44a346b2-4ddf-475b-9729-6e15d825198d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:55.812802 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:55.812770 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44a346b2-4ddf-475b-9729-6e15d825198d-metrics-certs" pod "44a346b2-4ddf-475b-9729-6e15d825198d" (UID: "44a346b2-4ddf-475b-9729-6e15d825198d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:55.890675 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:55.890638 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44a346b2-4ddf-475b-9729-6e15d825198d-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:55.890675 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:55.890679 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44a346b2-4ddf-475b-9729-6e15d825198d-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:55.890883 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:55.890696 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gt8m\" (UniqueName: \"kubernetes.io/projected/44a346b2-4ddf-475b-9729-6e15d825198d-kube-api-access-9gt8m\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:55.890883 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:55.890711 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a346b2-4ddf-475b-9729-6e15d825198d-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:29:56.585495 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:56.585460 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" Nov 02 18:29:56.585495 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:56.585475 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs" event={"ID":"44a346b2-4ddf-475b-9729-6e15d825198d","Type":"ContainerDied","Data":"19dd0a32babd23ce8d7ba32a01523cada321102c948fe064a256fef8968403f1"} Nov 02 18:29:56.586030 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:56.585529 5601 scope.go:122] "RemoveContainer" containerID="c89b85385248d06b13208691de04eacfef114e3ac310a94cfa2f76a655755d55" Nov 02 18:29:56.636669 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:56.635719 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs"] Nov 02 18:29:56.640396 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:56.640357 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-72f605-7595778566-7tqhs"] Nov 02 18:29:57.426087 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:29:57.426053 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a346b2-4ddf-475b-9729-6e15d825198d" path="/var/lib/kubelet/pods/44a346b2-4ddf-475b-9729-6e15d825198d/volumes" Nov 02 18:30:00.138133 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138098 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms"] Nov 02 18:30:00.138862 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138831 5601 cpu_manager.go:414] "containerMap: removing container" podUID="5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2" containerName="maas-api" Nov 02 18:30:00.138862 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138855 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2" containerName="maas-api" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138871 5601 cpu_manager.go:414] "containerMap: removing container" podUID="6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f" containerName="main" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138877 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f" containerName="main" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138889 5601 cpu_manager.go:414] "containerMap: removing container" podUID="2c9cb412-9cd3-444c-a7d0-ab4e7b397275" containerName="istio-proxy" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138894 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c9cb412-9cd3-444c-a7d0-ab4e7b397275" containerName="istio-proxy" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138906 5601 cpu_manager.go:414] "containerMap: removing container" podUID="d8fa2df6-995c-4f63-b100-8dcec94085d7" containerName="istio-proxy" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138911 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8fa2df6-995c-4f63-b100-8dcec94085d7" containerName="istio-proxy" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138920 5601 cpu_manager.go:414] "containerMap: removing container" podUID="7adb9d7e-827f-4573-a294-2681a27a79e0" containerName="maas-api" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138926 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7adb9d7e-827f-4573-a294-2681a27a79e0" containerName="maas-api" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138938 5601 cpu_manager.go:414] "containerMap: removing container" podUID="519549a8-7b5f-46f6-9e58-428c162bbe63" containerName="maas-api" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138946 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519549a8-7b5f-46f6-9e58-428c162bbe63" containerName="maas-api" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138957 5601 cpu_manager.go:414] "containerMap: removing container" podUID="575ac171-83b1-4d32-bc19-9a039f885c0a" containerName="istio-proxy" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138962 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575ac171-83b1-4d32-bc19-9a039f885c0a" containerName="istio-proxy" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138969 5601 cpu_manager.go:414] "containerMap: removing container" podUID="dc1c481d-b715-4ffd-8803-7371e21eead0" containerName="istio-proxy" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138973 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1c481d-b715-4ffd-8803-7371e21eead0" containerName="istio-proxy" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138982 5601 cpu_manager.go:414] "containerMap: removing container" podUID="993ea19b-d958-4efa-91c6-e9ba381b23ef" containerName="payload-processing" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138987 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993ea19b-d958-4efa-91c6-e9ba381b23ef" containerName="payload-processing" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138994 5601 cpu_manager.go:414] "containerMap: removing container" podUID="ea7a3828-ae1f-4f34-a847-3a03b824cd33" containerName="istio-proxy" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.138999 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea7a3828-ae1f-4f34-a847-3a03b824cd33" containerName="istio-proxy" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139005 5601 cpu_manager.go:414] "containerMap: removing container" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139011 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139021 5601 cpu_manager.go:414] "containerMap: removing container" podUID="6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed" containerName="payload-pre-processing" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139028 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed" containerName="payload-pre-processing" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139061 5601 cpu_manager.go:414] "containerMap: removing container" podUID="145f6bf1-b596-4e1e-a561-10c46c7c9b63" containerName="maas-api" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139067 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="145f6bf1-b596-4e1e-a561-10c46c7c9b63" containerName="maas-api" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139075 5601 cpu_manager.go:414] "containerMap: removing container" podUID="8fd72fbc-016a-43de-95f0-4d939b864aac" containerName="istio-proxy" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139080 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fd72fbc-016a-43de-95f0-4d939b864aac" containerName="istio-proxy" Nov 02 18:30:00.139072 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139086 5601 cpu_manager.go:414] "containerMap: removing container" podUID="c9f29635-35c4-419c-80e4-0239daa2b244" containerName="payload-processing" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139094 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f29635-35c4-419c-80e4-0239daa2b244" containerName="payload-processing" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139105 5601 cpu_manager.go:414] "containerMap: removing container" podUID="e5f3c081-92f7-4903-a974-779520012039" containerName="maas-api" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139110 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f3c081-92f7-4903-a974-779520012039" containerName="maas-api" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139116 5601 cpu_manager.go:414] "containerMap: removing container" podUID="ea7a3828-ae1f-4f34-a847-3a03b824cd33" containerName="istio-proxy" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139121 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea7a3828-ae1f-4f34-a847-3a03b824cd33" containerName="istio-proxy" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139131 5601 cpu_manager.go:414] "containerMap: removing container" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139135 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139147 5601 cpu_manager.go:414] "containerMap: removing container" podUID="0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" containerName="istio-proxy" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139152 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" containerName="istio-proxy" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139158 5601 cpu_manager.go:414] "containerMap: removing container" podUID="44a346b2-4ddf-475b-9729-6e15d825198d" containerName="maas-api" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139163 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a346b2-4ddf-475b-9729-6e15d825198d" containerName="maas-api" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139179 5601 cpu_manager.go:414] "containerMap: removing container" podUID="e6e6a7d0-45c4-4fce-aad6-edf7d92c2795" containerName="maas-api" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139185 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6e6a7d0-45c4-4fce-aad6-edf7d92c2795" containerName="maas-api" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139192 5601 cpu_manager.go:414] "containerMap: removing container" podUID="b98ac353-914e-4f05-b8b5-2c29eb3703d0" containerName="maas-api" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139197 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98ac353-914e-4f05-b8b5-2c29eb3703d0" containerName="maas-api" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139204 5601 cpu_manager.go:414] "containerMap: removing container" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139208 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139213 5601 cpu_manager.go:414] "containerMap: removing container" podUID="0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" containerName="istio-proxy" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139217 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" containerName="istio-proxy" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139301 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="7adb9d7e-827f-4573-a294-2681a27a79e0" containerName="maas-api" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139313 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="519549a8-7b5f-46f6-9e58-428c162bbe63" containerName="maas-api" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139318 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="575ac171-83b1-4d32-bc19-9a039f885c0a" containerName="istio-proxy" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139326 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1c481d-b715-4ffd-8803-7371e21eead0" containerName="istio-proxy" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139335 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="993ea19b-d958-4efa-91c6-e9ba381b23ef" containerName="payload-processing" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139343 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea7a3828-ae1f-4f34-a847-3a03b824cd33" containerName="istio-proxy" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139352 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="436816c2-2c00-4cad-aac7-298db2f5e8d5" containerName="main" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139358 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="6273dace-6c1a-4bb9-bcd6-3c4e0fbc62ed" containerName="payload-pre-processing" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139365 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="145f6bf1-b596-4e1e-a561-10c46c7c9b63" containerName="maas-api" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139372 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fd72fbc-016a-43de-95f0-4d939b864aac" containerName="istio-proxy" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139376 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f29635-35c4-419c-80e4-0239daa2b244" containerName="payload-processing" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139382 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f3c081-92f7-4903-a974-779520012039" containerName="maas-api" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139388 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea7a3828-ae1f-4f34-a847-3a03b824cd33" containerName="istio-proxy" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139394 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b3e335e-4328-429a-9433-728436384eba" containerName="main" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139410 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" containerName="istio-proxy" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139418 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a346b2-4ddf-475b-9729-6e15d825198d" containerName="maas-api" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139423 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6e6a7d0-45c4-4fce-aad6-edf7d92c2795" containerName="maas-api" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139429 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="b98ac353-914e-4f05-b8b5-2c29eb3703d0" containerName="maas-api" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139434 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d064f41-fece-402e-a07f-8515424b2c79" containerName="main" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139439 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d7f63eb-dd21-48bf-a0d5-2919ca7e668a" containerName="istio-proxy" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139444 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="5acd3d8f-7fcd-4a37-8e39-8e5ad4b835b2" containerName="maas-api" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139448 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e88c53c-c96d-4e18-9cf1-2d54e8f55f1f" containerName="main" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139455 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c9cb412-9cd3-444c-a7d0-ab4e7b397275" containerName="istio-proxy" Nov 02 18:30:00.140519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.139461 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8fa2df6-995c-4f63-b100-8dcec94085d7" containerName="istio-proxy" Nov 02 18:30:00.142969 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.142872 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" Nov 02 18:30:00.181674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.181053 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms"] Nov 02 18:30:00.181674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.181161 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9"] Nov 02 18:30:00.182043 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.182017 5601 cpu_manager.go:414] "containerMap: removing container" podUID="575ac171-83b1-4d32-bc19-9a039f885c0a" containerName="istio-proxy" Nov 02 18:30:00.182131 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.182045 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575ac171-83b1-4d32-bc19-9a039f885c0a" containerName="istio-proxy" Nov 02 18:30:00.182211 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.182194 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="575ac171-83b1-4d32-bc19-9a039f885c0a" containerName="istio-proxy" Nov 02 18:30:00.189663 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.186527 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4"] Nov 02 18:30:00.199298 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.195502 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" Nov 02 18:30:00.205710 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.205680 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9"] Nov 02 18:30:00.205886 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.205732 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4"] Nov 02 18:30:00.205886 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.205818 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" Nov 02 18:30:00.215997 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.215972 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\"" Nov 02 18:30:00.232696 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.225773 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6"] Nov 02 18:30:00.255933 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.255189 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" Nov 02 18:30:00.256118 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.255059 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q"] Nov 02 18:30:00.264520 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.263759 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6"] Nov 02 18:30:00.264520 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.263889 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" Nov 02 18:30:00.264520 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.264126 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q"] Nov 02 18:30:00.276290 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.269704 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01186ed8-9a15-4802-924a-85dd46e7dc8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms\" (UID: \"01186ed8-9a15-4802-924a-85dd46e7dc8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" Nov 02 18:30:00.276290 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.269765 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znzrt\" (UniqueName: \"kubernetes.io/projected/01186ed8-9a15-4802-924a-85dd46e7dc8b-kube-api-access-znzrt\") pod \"maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms\" (UID: \"01186ed8-9a15-4802-924a-85dd46e7dc8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" Nov 02 18:30:00.276290 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.269849 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5a3bb4-a17b-41cf-b512-f4de4bd0e471-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894070-7dck9\" (UID: \"1d5a3bb4-a17b-41cf-b512-f4de4bd0e471\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" Nov 02 18:30:00.276290 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.269911 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql9pj\" (UniqueName: \"kubernetes.io/projected/1d5a3bb4-a17b-41cf-b512-f4de4bd0e471-kube-api-access-ql9pj\") pod \"maas-api-key-cleanup-29894070-7dck9\" (UID: \"1d5a3bb4-a17b-41cf-b512-f4de4bd0e471\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" Nov 02 18:30:00.371174 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.371135 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5a3bb4-a17b-41cf-b512-f4de4bd0e471-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894070-7dck9\" (UID: \"1d5a3bb4-a17b-41cf-b512-f4de4bd0e471\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" Nov 02 18:30:00.371354 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.371185 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb3a82f-14e6-4cdf-8fa7-12db317f98f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q\" (UID: \"deb3a82f-14e6-4cdf-8fa7-12db317f98f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" Nov 02 18:30:00.371354 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.371242 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql9pj\" (UniqueName: \"kubernetes.io/projected/1d5a3bb4-a17b-41cf-b512-f4de4bd0e471-kube-api-access-ql9pj\") pod \"maas-api-key-cleanup-29894070-7dck9\" (UID: \"1d5a3bb4-a17b-41cf-b512-f4de4bd0e471\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" Nov 02 18:30:00.371495 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.371355 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6\" (UID: \"4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" Nov 02 18:30:00.371495 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.371404 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plwx5\" (UniqueName: \"kubernetes.io/projected/deb3a82f-14e6-4cdf-8fa7-12db317f98f8-kube-api-access-plwx5\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q\" (UID: \"deb3a82f-14e6-4cdf-8fa7-12db317f98f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" Nov 02 18:30:00.371675 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.371647 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01186ed8-9a15-4802-924a-85dd46e7dc8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms\" (UID: \"01186ed8-9a15-4802-924a-85dd46e7dc8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" Nov 02 18:30:00.371739 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.371725 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znzrt\" (UniqueName: \"kubernetes.io/projected/01186ed8-9a15-4802-924a-85dd46e7dc8b-kube-api-access-znzrt\") pod \"maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms\" (UID: \"01186ed8-9a15-4802-924a-85dd46e7dc8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" Nov 02 18:30:00.371789 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.371766 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac238df9-b947-47d0-a52a-38145c23f61a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4\" (UID: \"ac238df9-b947-47d0-a52a-38145c23f61a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" Nov 02 18:30:00.371841 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.371799 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d8tx\" (UniqueName: \"kubernetes.io/projected/ac238df9-b947-47d0-a52a-38145c23f61a-kube-api-access-8d8tx\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4\" (UID: \"ac238df9-b947-47d0-a52a-38145c23f61a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" Nov 02 18:30:00.371923 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.371904 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfdvw\" (UniqueName: \"kubernetes.io/projected/4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99-kube-api-access-dfdvw\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6\" (UID: \"4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" Nov 02 18:30:00.371979 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.371931 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5a3bb4-a17b-41cf-b512-f4de4bd0e471-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894070-7dck9\" (UID: \"1d5a3bb4-a17b-41cf-b512-f4de4bd0e471\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" Nov 02 18:30:00.372427 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.372400 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01186ed8-9a15-4802-924a-85dd46e7dc8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms\" (UID: \"01186ed8-9a15-4802-924a-85dd46e7dc8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" Nov 02 18:30:00.383531 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.383495 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql9pj\" (UniqueName: \"kubernetes.io/projected/1d5a3bb4-a17b-41cf-b512-f4de4bd0e471-kube-api-access-ql9pj\") pod \"maas-api-key-cleanup-29894070-7dck9\" (UID: \"1d5a3bb4-a17b-41cf-b512-f4de4bd0e471\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" Nov 02 18:30:00.383838 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.383662 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znzrt\" (UniqueName: \"kubernetes.io/projected/01186ed8-9a15-4802-924a-85dd46e7dc8b-kube-api-access-znzrt\") pod \"maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms\" (UID: \"01186ed8-9a15-4802-924a-85dd46e7dc8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" Nov 02 18:30:00.407486 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.403099 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r"] Nov 02 18:30:00.410707 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.410677 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:00.411106 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.411082 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r"] Nov 02 18:30:00.418239 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.418213 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-7771bf\"/\"default-dockercfg-fl2z6\"" Nov 02 18:30:00.418467 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.418248 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-7771bf\"/\"sub-test-model-7771bf-kserve-self-signed-certs\"" Nov 02 18:30:00.418467 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.418309 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-7771bf\"/\"kube-root-ca.crt\"" Nov 02 18:30:00.418467 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.418399 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-7771bf\"/\"openshift-service-ca.crt\"" Nov 02 18:30:00.473714 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.473668 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6\" (UID: \"4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" Nov 02 18:30:00.473976 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.473742 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plwx5\" (UniqueName: \"kubernetes.io/projected/deb3a82f-14e6-4cdf-8fa7-12db317f98f8-kube-api-access-plwx5\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q\" (UID: \"deb3a82f-14e6-4cdf-8fa7-12db317f98f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" Nov 02 18:30:00.473976 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.473896 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac238df9-b947-47d0-a52a-38145c23f61a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4\" (UID: \"ac238df9-b947-47d0-a52a-38145c23f61a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" Nov 02 18:30:00.473976 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.473938 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d8tx\" (UniqueName: \"kubernetes.io/projected/ac238df9-b947-47d0-a52a-38145c23f61a-kube-api-access-8d8tx\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4\" (UID: \"ac238df9-b947-47d0-a52a-38145c23f61a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" Nov 02 18:30:00.473976 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.473973 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfdvw\" (UniqueName: \"kubernetes.io/projected/4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99-kube-api-access-dfdvw\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6\" (UID: \"4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" Nov 02 18:30:00.474226 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.474060 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb3a82f-14e6-4cdf-8fa7-12db317f98f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q\" (UID: \"deb3a82f-14e6-4cdf-8fa7-12db317f98f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" Nov 02 18:30:00.475097 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.475073 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb3a82f-14e6-4cdf-8fa7-12db317f98f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q\" (UID: \"deb3a82f-14e6-4cdf-8fa7-12db317f98f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" Nov 02 18:30:00.476912 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.476856 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac238df9-b947-47d0-a52a-38145c23f61a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4\" (UID: \"ac238df9-b947-47d0-a52a-38145c23f61a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" Nov 02 18:30:00.477905 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.477874 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6\" (UID: \"4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" Nov 02 18:30:00.496647 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.494397 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plwx5\" (UniqueName: \"kubernetes.io/projected/deb3a82f-14e6-4cdf-8fa7-12db317f98f8-kube-api-access-plwx5\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q\" (UID: \"deb3a82f-14e6-4cdf-8fa7-12db317f98f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" Nov 02 18:30:00.502168 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.500430 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfdvw\" (UniqueName: \"kubernetes.io/projected/4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99-kube-api-access-dfdvw\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6\" (UID: \"4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" Nov 02 18:30:00.508329 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.508302 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" Nov 02 18:30:00.513680 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.511837 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d8tx\" (UniqueName: \"kubernetes.io/projected/ac238df9-b947-47d0-a52a-38145c23f61a-kube-api-access-8d8tx\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4\" (UID: \"ac238df9-b947-47d0-a52a-38145c23f61a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" Nov 02 18:30:00.536201 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.536164 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" Nov 02 18:30:00.550675 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.550643 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" Nov 02 18:30:00.577127 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.575524 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d098a18f-93d8-4958-833b-252443a3b8b8-tls-certs\") pod \"sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:00.577127 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.575685 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-model-cache\") pod \"sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:00.577127 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.575730 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjl78\" (UniqueName: \"kubernetes.io/projected/d098a18f-93d8-4958-833b-252443a3b8b8-kube-api-access-kjl78\") pod \"sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:00.577127 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.575769 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-tmp-dir\") pod \"sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:00.577127 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.575808 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-home\") pod \"sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:00.577127 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.575834 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-dshm\") pod \"sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:00.601203 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.601169 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" Nov 02 18:30:00.606289 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.606054 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" Nov 02 18:30:00.789512 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.789470 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-home\") pod \"sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:00.789665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.789523 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-dshm\") pod \"sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:00.789745 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.789673 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d098a18f-93d8-4958-833b-252443a3b8b8-tls-certs\") pod \"sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:00.789808 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.789781 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-model-cache\") pod \"sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:00.789863 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.789822 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjl78\" (UniqueName: \"kubernetes.io/projected/d098a18f-93d8-4958-833b-252443a3b8b8-kube-api-access-kjl78\") pod \"sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:00.789920 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.789864 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-tmp-dir\") pod \"sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:00.790301 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.790268 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-tmp-dir\") pod \"sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:00.790876 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.790846 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-home\") pod \"sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:00.793545 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.793518 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-dshm\") pod \"sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:00.836685 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.799035 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-model-cache\") pod \"sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:00.869935 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.869587 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjl78\" (UniqueName: \"kubernetes.io/projected/d098a18f-93d8-4958-833b-252443a3b8b8-kube-api-access-kjl78\") pod \"sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:00.885347 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:00.885280 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d098a18f-93d8-4958-833b-252443a3b8b8-tls-certs\") pod \"sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:01.011011 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:01.010978 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms"] Nov 02 18:30:01.016110 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:30:01.016078 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01186ed8_9a15_4802_924a_85dd46e7dc8b.slice/crio-22a431637de0d25e0f4fed359566e08947feb25ac8227fe88278ea9f1ca11ca8 WatchSource:0}: Error finding container 22a431637de0d25e0f4fed359566e08947feb25ac8227fe88278ea9f1ca11ca8: Status 404 returned error can't find the container with id 22a431637de0d25e0f4fed359566e08947feb25ac8227fe88278ea9f1ca11ca8 Nov 02 18:30:01.063752 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:01.063727 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9"] Nov 02 18:30:01.066122 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:30:01.066083 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5a3bb4_a17b_41cf_b512_f4de4bd0e471.slice/crio-cbd10501bb9b3ad60f6638275848ac40ac5a0214cc3066c74d5037856297a535 WatchSource:0}: Error finding container cbd10501bb9b3ad60f6638275848ac40ac5a0214cc3066c74d5037856297a535: Status 404 returned error can't find the container with id cbd10501bb9b3ad60f6638275848ac40ac5a0214cc3066c74d5037856297a535 Nov 02 18:30:01.081479 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:01.081453 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:01.104407 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:01.104372 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4"] Nov 02 18:30:01.104605 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:30:01.104371 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac238df9_b947_47d0_a52a_38145c23f61a.slice/crio-ff716db9d2ad8311d8bb53b0dd897dd414307f8716fe675cafaa9bad68312f85 WatchSource:0}: Error finding container ff716db9d2ad8311d8bb53b0dd897dd414307f8716fe675cafaa9bad68312f85: Status 404 returned error can't find the container with id ff716db9d2ad8311d8bb53b0dd897dd414307f8716fe675cafaa9bad68312f85 Nov 02 18:30:01.179840 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:01.179607 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6"] Nov 02 18:30:01.184919 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:01.184887 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q"] Nov 02 18:30:01.382948 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:01.382924 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r"] Nov 02 18:30:01.703364 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:01.703330 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" event={"ID":"01186ed8-9a15-4802-924a-85dd46e7dc8b","Type":"ContainerStarted","Data":"b7713c0bd062fa7fbafa2cb4e56e24d9b4d00fd6f166b4c34e6d37a8831b9ced"} Nov 02 18:30:01.703522 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:01.703406 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" event={"ID":"01186ed8-9a15-4802-924a-85dd46e7dc8b","Type":"ContainerStarted","Data":"22a431637de0d25e0f4fed359566e08947feb25ac8227fe88278ea9f1ca11ca8"} Nov 02 18:30:01.705259 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:01.705228 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" event={"ID":"deb3a82f-14e6-4cdf-8fa7-12db317f98f8","Type":"ContainerStarted","Data":"d26fa7f94ce1717f1e47457bf0101484e0e83e32fff777ba9f1a048d2e5eb2fc"} Nov 02 18:30:01.707356 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:01.707261 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" event={"ID":"1d5a3bb4-a17b-41cf-b512-f4de4bd0e471","Type":"ContainerStarted","Data":"50a3f350994f30febdb4fefacd73a1e04c47d819a67e1a6ea3726798b9ae217c"} Nov 02 18:30:01.707356 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:01.707308 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" event={"ID":"1d5a3bb4-a17b-41cf-b512-f4de4bd0e471","Type":"ContainerStarted","Data":"cbd10501bb9b3ad60f6638275848ac40ac5a0214cc3066c74d5037856297a535"} Nov 02 18:30:01.709032 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:01.709007 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" event={"ID":"ac238df9-b947-47d0-a52a-38145c23f61a","Type":"ContainerStarted","Data":"9138d177113e338c24ba0f457e2b036e834e285930ee363454e0550213a334b2"} Nov 02 18:30:01.709142 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:01.709051 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" event={"ID":"ac238df9-b947-47d0-a52a-38145c23f61a","Type":"ContainerStarted","Data":"ff716db9d2ad8311d8bb53b0dd897dd414307f8716fe675cafaa9bad68312f85"} Nov 02 18:30:01.710553 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:01.710533 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" event={"ID":"4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99","Type":"ContainerStarted","Data":"96082e492c1c1374988ede5a5a4c7dbadd6b78e4d8e4d7d6c78aa5383482d69b"} Nov 02 18:30:01.712007 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:01.711985 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" event={"ID":"d098a18f-93d8-4958-833b-252443a3b8b8","Type":"ContainerStarted","Data":"e9014e926473b0c77c2a8233638476f7486cebbdbdc564534e9bc2a0d71f8643"} Nov 02 18:30:01.719596 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:01.719545 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" podStartSLOduration=1.719530969 podStartE2EDuration="1.719530969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:01.718925046 +0000 UTC m=+3366.409979058" watchObservedRunningTime="2026-11-02 18:30:01.719530969 +0000 UTC m=+3366.410584981" Nov 02 18:30:01.761325 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:01.761278 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" podStartSLOduration=1.761264484 podStartE2EDuration="1.761264484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:01.76096923 +0000 UTC m=+3366.452023241" watchObservedRunningTime="2026-11-02 18:30:01.761264484 +0000 UTC m=+3366.452318497" Nov 02 18:30:01.765350 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:01.765073 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" podStartSLOduration=1.765060632 podStartE2EDuration="1.765060632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:01.74093677 +0000 UTC m=+3366.431990785" watchObservedRunningTime="2026-11-02 18:30:01.765060632 +0000 UTC m=+3366.456114647" Nov 02 18:30:02.720325 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:02.720288 5601 generic.go:356] "Generic (PLEG): container finished" podID="01186ed8-9a15-4802-924a-85dd46e7dc8b" containerID="b7713c0bd062fa7fbafa2cb4e56e24d9b4d00fd6f166b4c34e6d37a8831b9ced" exitCode=60 Nov 02 18:30:02.720770 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:02.720374 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" event={"ID":"01186ed8-9a15-4802-924a-85dd46e7dc8b","Type":"ContainerDied","Data":"b7713c0bd062fa7fbafa2cb4e56e24d9b4d00fd6f166b4c34e6d37a8831b9ced"} Nov 02 18:30:02.720770 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:02.720697 5601 scope.go:122] "RemoveContainer" containerID="b7713c0bd062fa7fbafa2cb4e56e24d9b4d00fd6f166b4c34e6d37a8831b9ced" Nov 02 18:30:02.722117 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:02.722092 5601 generic.go:356] "Generic (PLEG): container finished" podID="deb3a82f-14e6-4cdf-8fa7-12db317f98f8" containerID="96cd3289e670bac2d21dfce114bdb726dc6c77a375e369b7b986b4496a6acee9" exitCode=60 Nov 02 18:30:02.722251 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:02.722153 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" event={"ID":"deb3a82f-14e6-4cdf-8fa7-12db317f98f8","Type":"ContainerDied","Data":"96cd3289e670bac2d21dfce114bdb726dc6c77a375e369b7b986b4496a6acee9"} Nov 02 18:30:02.722414 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:02.722364 5601 scope.go:122] "RemoveContainer" containerID="96cd3289e670bac2d21dfce114bdb726dc6c77a375e369b7b986b4496a6acee9" Nov 02 18:30:02.725190 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:02.725158 5601 generic.go:356] "Generic (PLEG): container finished" podID="1d5a3bb4-a17b-41cf-b512-f4de4bd0e471" containerID="50a3f350994f30febdb4fefacd73a1e04c47d819a67e1a6ea3726798b9ae217c" exitCode=60 Nov 02 18:30:02.725292 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:02.725258 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" event={"ID":"1d5a3bb4-a17b-41cf-b512-f4de4bd0e471","Type":"ContainerDied","Data":"50a3f350994f30febdb4fefacd73a1e04c47d819a67e1a6ea3726798b9ae217c"} Nov 02 18:30:02.725499 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:02.725464 5601 scope.go:122] "RemoveContainer" containerID="50a3f350994f30febdb4fefacd73a1e04c47d819a67e1a6ea3726798b9ae217c" Nov 02 18:30:02.727422 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:02.727400 5601 generic.go:356] "Generic (PLEG): container finished" podID="ac238df9-b947-47d0-a52a-38145c23f61a" containerID="9138d177113e338c24ba0f457e2b036e834e285930ee363454e0550213a334b2" exitCode=60 Nov 02 18:30:02.727497 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:02.727475 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" event={"ID":"ac238df9-b947-47d0-a52a-38145c23f61a","Type":"ContainerDied","Data":"9138d177113e338c24ba0f457e2b036e834e285930ee363454e0550213a334b2"} Nov 02 18:30:02.727690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:02.727675 5601 scope.go:122] "RemoveContainer" containerID="9138d177113e338c24ba0f457e2b036e834e285930ee363454e0550213a334b2" Nov 02 18:30:02.729089 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:02.729067 5601 generic.go:356] "Generic (PLEG): container finished" podID="4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99" containerID="ee7d13908c2d4da90c5e0fe9e6c489375e5a2551625ede24920e9304d258798d" exitCode=60 Nov 02 18:30:02.729194 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:02.729143 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" event={"ID":"4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99","Type":"ContainerDied","Data":"ee7d13908c2d4da90c5e0fe9e6c489375e5a2551625ede24920e9304d258798d"} Nov 02 18:30:02.729344 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:02.729328 5601 scope.go:122] "RemoveContainer" containerID="ee7d13908c2d4da90c5e0fe9e6c489375e5a2551625ede24920e9304d258798d" Nov 02 18:30:02.730718 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:02.730698 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" event={"ID":"d098a18f-93d8-4958-833b-252443a3b8b8","Type":"ContainerStarted","Data":"47d38c97c644a46e25664e679daf090967ac4502249994f273c83de5e81b929f"} Nov 02 18:30:02.779098 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:02.779038 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" podStartSLOduration=2.779020359 podStartE2EDuration="2.779020359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:02.776096147 +0000 UTC m=+3367.467150160" watchObservedRunningTime="2026-11-02 18:30:02.779020359 +0000 UTC m=+3367.470074371" Nov 02 18:30:03.082170 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:03.082134 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:03.750235 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:03.749247 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" event={"ID":"01186ed8-9a15-4802-924a-85dd46e7dc8b","Type":"ContainerStarted","Data":"52bfd3200ed0bb00952d9d2a3b7fb9e4968b0eb7abeac928dead17380230f2f7"} Nov 02 18:30:03.757402 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:03.756651 5601 scope.go:122] "RemoveContainer" containerID="d10ffc4d490fa29217f4031ec14f258775e90a3273f14fde6307527b8b710960" Nov 02 18:30:03.757402 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:03.757135 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q_odh-ai-gateway-infra(deb3a82f-14e6-4cdf-8fa7-12db317f98f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" podUID="deb3a82f-14e6-4cdf-8fa7-12db317f98f8" Nov 02 18:30:03.757602 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:03.757469 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" event={"ID":"deb3a82f-14e6-4cdf-8fa7-12db317f98f8","Type":"ContainerStarted","Data":"d10ffc4d490fa29217f4031ec14f258775e90a3273f14fde6307527b8b710960"} Nov 02 18:30:03.769455 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:03.769373 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" event={"ID":"1d5a3bb4-a17b-41cf-b512-f4de4bd0e471","Type":"ContainerStarted","Data":"6b7d8c36bb39fe4b2638d01e6ca88236f87e143563147489e9786b6e8dbfd093"} Nov 02 18:30:03.774548 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:03.774516 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" event={"ID":"ac238df9-b947-47d0-a52a-38145c23f61a","Type":"ContainerStarted","Data":"aa37258f1e1d1d74e8509017ddc82a8de7cb2206a7321f448bbb9af211e14627"} Nov 02 18:30:03.777124 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:03.777100 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" event={"ID":"4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99","Type":"ContainerStarted","Data":"dbfa2073ccb506ce0485749b9a7afbca754e72727b78ba2bb54657e6d2f7201d"} Nov 02 18:30:03.777387 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:03.777371 5601 scope.go:122] "RemoveContainer" containerID="dbfa2073ccb506ce0485749b9a7afbca754e72727b78ba2bb54657e6d2f7201d" Nov 02 18:30:03.777610 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:03.777592 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6_odh-ai-gateway-infra(4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" podUID="4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99" Nov 02 18:30:04.398959 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.398869 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6"] Nov 02 18:30:04.486564 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:04.486524 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:30:04.785515 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.785481 5601 generic.go:356] "Generic (PLEG): container finished" podID="ac238df9-b947-47d0-a52a-38145c23f61a" containerID="aa37258f1e1d1d74e8509017ddc82a8de7cb2206a7321f448bbb9af211e14627" exitCode=60 Nov 02 18:30:04.786020 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.785583 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" event={"ID":"ac238df9-b947-47d0-a52a-38145c23f61a","Type":"ContainerDied","Data":"aa37258f1e1d1d74e8509017ddc82a8de7cb2206a7321f448bbb9af211e14627"} Nov 02 18:30:04.786020 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.785681 5601 scope.go:122] "RemoveContainer" containerID="9138d177113e338c24ba0f457e2b036e834e285930ee363454e0550213a334b2" Nov 02 18:30:04.786130 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.786025 5601 scope.go:122] "RemoveContainer" containerID="aa37258f1e1d1d74e8509017ddc82a8de7cb2206a7321f448bbb9af211e14627" Nov 02 18:30:04.786369 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:04.786344 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4_odh-ai-gateway-infra(ac238df9-b947-47d0-a52a-38145c23f61a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" podUID="ac238df9-b947-47d0-a52a-38145c23f61a" Nov 02 18:30:04.787720 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.787680 5601 generic.go:356] "Generic (PLEG): container finished" podID="4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99" containerID="dbfa2073ccb506ce0485749b9a7afbca754e72727b78ba2bb54657e6d2f7201d" exitCode=60 Nov 02 18:30:04.787813 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.787752 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" event={"ID":"4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99","Type":"ContainerDied","Data":"dbfa2073ccb506ce0485749b9a7afbca754e72727b78ba2bb54657e6d2f7201d"} Nov 02 18:30:04.788050 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.788031 5601 scope.go:122] "RemoveContainer" containerID="dbfa2073ccb506ce0485749b9a7afbca754e72727b78ba2bb54657e6d2f7201d" Nov 02 18:30:04.788326 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:04.788308 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6_odh-ai-gateway-infra(4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" podUID="4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99" Nov 02 18:30:04.789710 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.789682 5601 generic.go:356] "Generic (PLEG): container finished" podID="01186ed8-9a15-4802-924a-85dd46e7dc8b" containerID="52bfd3200ed0bb00952d9d2a3b7fb9e4968b0eb7abeac928dead17380230f2f7" exitCode=60 Nov 02 18:30:04.789799 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.789761 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" event={"ID":"01186ed8-9a15-4802-924a-85dd46e7dc8b","Type":"ContainerDied","Data":"52bfd3200ed0bb00952d9d2a3b7fb9e4968b0eb7abeac928dead17380230f2f7"} Nov 02 18:30:04.789962 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.789947 5601 scope.go:122] "RemoveContainer" containerID="52bfd3200ed0bb00952d9d2a3b7fb9e4968b0eb7abeac928dead17380230f2f7" Nov 02 18:30:04.790227 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:04.790197 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms_odh-ai-gateway-infra(01186ed8-9a15-4802-924a-85dd46e7dc8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" podUID="01186ed8-9a15-4802-924a-85dd46e7dc8b" Nov 02 18:30:04.791676 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.791653 5601 generic.go:356] "Generic (PLEG): container finished" podID="deb3a82f-14e6-4cdf-8fa7-12db317f98f8" containerID="d10ffc4d490fa29217f4031ec14f258775e90a3273f14fde6307527b8b710960" exitCode=60 Nov 02 18:30:04.791766 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.791725 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" event={"ID":"deb3a82f-14e6-4cdf-8fa7-12db317f98f8","Type":"ContainerDied","Data":"d10ffc4d490fa29217f4031ec14f258775e90a3273f14fde6307527b8b710960"} Nov 02 18:30:04.791979 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.791960 5601 scope.go:122] "RemoveContainer" containerID="d10ffc4d490fa29217f4031ec14f258775e90a3273f14fde6307527b8b710960" Nov 02 18:30:04.792252 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:04.792233 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q_odh-ai-gateway-infra(deb3a82f-14e6-4cdf-8fa7-12db317f98f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" podUID="deb3a82f-14e6-4cdf-8fa7-12db317f98f8" Nov 02 18:30:04.793580 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.793560 5601 generic.go:356] "Generic (PLEG): container finished" podID="1d5a3bb4-a17b-41cf-b512-f4de4bd0e471" containerID="6b7d8c36bb39fe4b2638d01e6ca88236f87e143563147489e9786b6e8dbfd093" exitCode=60 Nov 02 18:30:04.793707 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.793644 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" event={"ID":"1d5a3bb4-a17b-41cf-b512-f4de4bd0e471","Type":"ContainerDied","Data":"6b7d8c36bb39fe4b2638d01e6ca88236f87e143563147489e9786b6e8dbfd093"} Nov 02 18:30:04.793863 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.793843 5601 scope.go:122] "RemoveContainer" containerID="6b7d8c36bb39fe4b2638d01e6ca88236f87e143563147489e9786b6e8dbfd093" Nov 02 18:30:04.794132 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:04.794108 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894070-7dck9_odh-ai-gateway-infra(1d5a3bb4-a17b-41cf-b512-f4de4bd0e471)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" podUID="1d5a3bb4-a17b-41cf-b512-f4de4bd0e471" Nov 02 18:30:04.803335 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.803218 5601 scope.go:122] "RemoveContainer" containerID="ee7d13908c2d4da90c5e0fe9e6c489375e5a2551625ede24920e9304d258798d" Nov 02 18:30:04.818710 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.818690 5601 scope.go:122] "RemoveContainer" containerID="b7713c0bd062fa7fbafa2cb4e56e24d9b4d00fd6f166b4c34e6d37a8831b9ced" Nov 02 18:30:04.834552 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.834531 5601 scope.go:122] "RemoveContainer" containerID="96cd3289e670bac2d21dfce114bdb726dc6c77a375e369b7b986b4496a6acee9" Nov 02 18:30:04.849709 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:04.849684 5601 scope.go:122] "RemoveContainer" containerID="50a3f350994f30febdb4fefacd73a1e04c47d819a67e1a6ea3726798b9ae217c" Nov 02 18:30:05.805899 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:05.805868 5601 scope.go:122] "RemoveContainer" containerID="d10ffc4d490fa29217f4031ec14f258775e90a3273f14fde6307527b8b710960" Nov 02 18:30:05.806312 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:05.806249 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q_odh-ai-gateway-infra(deb3a82f-14e6-4cdf-8fa7-12db317f98f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" podUID="deb3a82f-14e6-4cdf-8fa7-12db317f98f8" Nov 02 18:30:05.807566 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:05.807542 5601 scope.go:122] "RemoveContainer" containerID="6b7d8c36bb39fe4b2638d01e6ca88236f87e143563147489e9786b6e8dbfd093" Nov 02 18:30:05.807978 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:05.807930 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894070-7dck9_odh-ai-gateway-infra(1d5a3bb4-a17b-41cf-b512-f4de4bd0e471)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" podUID="1d5a3bb4-a17b-41cf-b512-f4de4bd0e471" Nov 02 18:30:05.809420 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:05.809397 5601 scope.go:122] "RemoveContainer" containerID="aa37258f1e1d1d74e8509017ddc82a8de7cb2206a7321f448bbb9af211e14627" Nov 02 18:30:05.809724 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:05.809697 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4_odh-ai-gateway-infra(ac238df9-b947-47d0-a52a-38145c23f61a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" podUID="ac238df9-b947-47d0-a52a-38145c23f61a" Nov 02 18:30:05.811087 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:05.811069 5601 scope.go:122] "RemoveContainer" containerID="dbfa2073ccb506ce0485749b9a7afbca754e72727b78ba2bb54657e6d2f7201d" Nov 02 18:30:05.811323 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:05.811304 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6_odh-ai-gateway-infra(4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" podUID="4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99" Nov 02 18:30:05.813552 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:05.813534 5601 scope.go:122] "RemoveContainer" containerID="52bfd3200ed0bb00952d9d2a3b7fb9e4968b0eb7abeac928dead17380230f2f7" Nov 02 18:30:05.813790 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:05.813771 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms_odh-ai-gateway-infra(01186ed8-9a15-4802-924a-85dd46e7dc8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" podUID="01186ed8-9a15-4802-924a-85dd46e7dc8b" Nov 02 18:30:10.127515 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:10.117695 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55"] Nov 02 18:30:10.127515 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:10.117981 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" podUID="ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" containerName="istio-proxy" containerID="cri-o://82fa031de84b14f2d407f7d5b7d75fb892999bff35e8220916dd4fd1b0f5c856" gracePeriod=30 Nov 02 18:30:11.081829 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:11.081790 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:11.094980 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:11.094951 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:11.879960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:11.879926 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:30:12.945198 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:12.945160 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg"] Nov 02 18:30:14.431570 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:14.420227 5601 scope.go:122] "RemoveContainer" containerID="6b7d8c36bb39fe4b2638d01e6ca88236f87e143563147489e9786b6e8dbfd093" Nov 02 18:30:14.892281 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:14.892245 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" event={"ID":"1d5a3bb4-a17b-41cf-b512-f4de4bd0e471","Type":"ContainerStarted","Data":"57aab209da16b636d7c955bd00fc2e9a1d05c70a4be453cfd96e05a0b0c406d2"} Nov 02 18:30:15.420303 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.420264 5601 scope.go:122] "RemoveContainer" containerID="aa37258f1e1d1d74e8509017ddc82a8de7cb2206a7321f448bbb9af211e14627" Nov 02 18:30:15.423674 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.423409 5601 scope.go:122] "RemoveContainer" containerID="d10ffc4d490fa29217f4031ec14f258775e90a3273f14fde6307527b8b710960" Nov 02 18:30:15.730224 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.729820 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 18:30:15.846081 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.846039 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-data\") pod \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " Nov 02 18:30:15.846283 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.846118 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-envoy\") pod \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " Nov 02 18:30:15.846283 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.846173 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-credential-socket\") pod \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " Nov 02 18:30:15.846283 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.846241 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-workload-certs\") pod \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " Nov 02 18:30:15.846495 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.846294 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-token\" (UniqueName: \"kubernetes.io/projected/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-token\") pod \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " Nov 02 18:30:15.846495 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.846331 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-workload-socket\") pod \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " Nov 02 18:30:15.846495 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.846377 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istiod-ca-cert\") pod \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " Nov 02 18:30:15.846495 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.846456 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-kube-api-access-rfkzx\" (UniqueName: \"kubernetes.io/projected/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-kube-api-access-rfkzx\") pod \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " Nov 02 18:30:15.846753 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.846541 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-podinfo\") pod \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\" (UID: \"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda\") " Nov 02 18:30:15.852062 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.848074 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-workload-certs" pod "ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" (UID: "ca8be970-ee6a-4fbe-94b1-c63f75fa4fda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:15.852062 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.848303 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-data" pod "ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" (UID: "ca8be970-ee6a-4fbe-94b1-c63f75fa4fda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:15.852062 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.848781 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-credential-socket" pod "ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" (UID: "ca8be970-ee6a-4fbe-94b1-c63f75fa4fda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:15.852062 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.848956 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-workload-socket" pod "ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" (UID: "ca8be970-ee6a-4fbe-94b1-c63f75fa4fda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:15.852062 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.849749 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istiod-ca-cert" pod "ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" (UID: "ca8be970-ee6a-4fbe-94b1-c63f75fa4fda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:15.922856 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.920962 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-token" pod "ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" (UID: "ca8be970-ee6a-4fbe-94b1-c63f75fa4fda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:15.922856 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.921133 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-envoy" pod "ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" (UID: "ca8be970-ee6a-4fbe-94b1-c63f75fa4fda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:15.929115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.929063 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-kube-api-access-rfkzx" pod "ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" (UID: "ca8be970-ee6a-4fbe-94b1-c63f75fa4fda"). InnerVolumeSpecName "kube-api-access-rfkzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:15.936472 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.934102 5601 generic.go:356] "Generic (PLEG): container finished" podID="1d5a3bb4-a17b-41cf-b512-f4de4bd0e471" containerID="57aab209da16b636d7c955bd00fc2e9a1d05c70a4be453cfd96e05a0b0c406d2" exitCode=60 Nov 02 18:30:15.936472 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.934279 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" event={"ID":"1d5a3bb4-a17b-41cf-b512-f4de4bd0e471","Type":"ContainerDied","Data":"57aab209da16b636d7c955bd00fc2e9a1d05c70a4be453cfd96e05a0b0c406d2"} Nov 02 18:30:15.936472 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.934340 5601 scope.go:122] "RemoveContainer" containerID="6b7d8c36bb39fe4b2638d01e6ca88236f87e143563147489e9786b6e8dbfd093" Nov 02 18:30:15.936472 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.934901 5601 scope.go:122] "RemoveContainer" containerID="57aab209da16b636d7c955bd00fc2e9a1d05c70a4be453cfd96e05a0b0c406d2" Nov 02 18:30:15.936472 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:15.935268 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894070-7dck9_odh-ai-gateway-infra(1d5a3bb4-a17b-41cf-b512-f4de4bd0e471)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" podUID="1d5a3bb4-a17b-41cf-b512-f4de4bd0e471" Nov 02 18:30:15.936472 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.935689 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-podinfo" pod "ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" (UID: "ca8be970-ee6a-4fbe-94b1-c63f75fa4fda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:30:15.959295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.941858 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9"] Nov 02 18:30:15.959295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.948480 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:15.959295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.948508 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfkzx\" (UniqueName: \"kubernetes.io/projected/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-kube-api-access-rfkzx\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:15.959295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.948525 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:15.959295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.948539 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:15.959295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.948553 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:15.959295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.948566 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:15.959295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.948580 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:15.959295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.948594 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:15.959295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.948607 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:15.959295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.953945 5601 generic.go:356] "Generic (PLEG): container finished" podID="ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" containerID="82fa031de84b14f2d407f7d5b7d75fb892999bff35e8220916dd4fd1b0f5c856" exitCode=0 Nov 02 18:30:15.959295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.954058 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" event={"ID":"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda","Type":"ContainerDied","Data":"82fa031de84b14f2d407f7d5b7d75fb892999bff35e8220916dd4fd1b0f5c856"} Nov 02 18:30:15.959295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.954105 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" event={"ID":"ca8be970-ee6a-4fbe-94b1-c63f75fa4fda","Type":"ContainerDied","Data":"d6e9b42e6e1a13e7be3b5452aebc2a5471750bdf97e4c74b20d6371ad80778ee"} Nov 02 18:30:15.959295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.954189 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55" Nov 02 18:30:15.982559 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.982530 5601 scope.go:122] "RemoveContainer" containerID="82fa031de84b14f2d407f7d5b7d75fb892999bff35e8220916dd4fd1b0f5c856" Nov 02 18:30:15.992169 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.992134 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55"] Nov 02 18:30:15.997804 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.997775 5601 scope.go:122] "RemoveContainer" containerID="82fa031de84b14f2d407f7d5b7d75fb892999bff35e8220916dd4fd1b0f5c856" Nov 02 18:30:15.998173 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:15.998145 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82fa031de84b14f2d407f7d5b7d75fb892999bff35e8220916dd4fd1b0f5c856\": container with ID starting with 82fa031de84b14f2d407f7d5b7d75fb892999bff35e8220916dd4fd1b0f5c856 not found: ID does not exist" containerID="82fa031de84b14f2d407f7d5b7d75fb892999bff35e8220916dd4fd1b0f5c856" Nov 02 18:30:15.998254 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.998187 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82fa031de84b14f2d407f7d5b7d75fb892999bff35e8220916dd4fd1b0f5c856"} err="failed to get container status \"82fa031de84b14f2d407f7d5b7d75fb892999bff35e8220916dd4fd1b0f5c856\": rpc error: code = NotFound desc = could not find container \"82fa031de84b14f2d407f7d5b7d75fb892999bff35e8220916dd4fd1b0f5c856\": container with ID starting with 82fa031de84b14f2d407f7d5b7d75fb892999bff35e8220916dd4fd1b0f5c856 not found: ID does not exist" Nov 02 18:30:15.998254 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:15.998243 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bzb55"] Nov 02 18:30:16.964473 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:16.964436 5601 generic.go:356] "Generic (PLEG): container finished" podID="deb3a82f-14e6-4cdf-8fa7-12db317f98f8" containerID="23aefc63bd420d65c4d4edce208881567fb36c8924593d9a8415fc63d78fceb1" exitCode=60 Nov 02 18:30:16.965002 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:16.964507 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" event={"ID":"deb3a82f-14e6-4cdf-8fa7-12db317f98f8","Type":"ContainerDied","Data":"23aefc63bd420d65c4d4edce208881567fb36c8924593d9a8415fc63d78fceb1"} Nov 02 18:30:16.965002 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:16.964597 5601 scope.go:122] "RemoveContainer" containerID="d10ffc4d490fa29217f4031ec14f258775e90a3273f14fde6307527b8b710960" Nov 02 18:30:16.965131 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:16.965028 5601 scope.go:122] "RemoveContainer" containerID="23aefc63bd420d65c4d4edce208881567fb36c8924593d9a8415fc63d78fceb1" Nov 02 18:30:16.965342 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:16.965322 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q_odh-ai-gateway-infra(deb3a82f-14e6-4cdf-8fa7-12db317f98f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" podUID="deb3a82f-14e6-4cdf-8fa7-12db317f98f8" Nov 02 18:30:16.969525 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:16.969502 5601 generic.go:356] "Generic (PLEG): container finished" podID="ac238df9-b947-47d0-a52a-38145c23f61a" containerID="397376842bf6deee1e38fadd7542309070a46d6c7496158c7df3b26c92172c7b" exitCode=60 Nov 02 18:30:16.969797 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:16.969542 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" event={"ID":"ac238df9-b947-47d0-a52a-38145c23f61a","Type":"ContainerDied","Data":"397376842bf6deee1e38fadd7542309070a46d6c7496158c7df3b26c92172c7b"} Nov 02 18:30:16.969990 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:16.969968 5601 scope.go:122] "RemoveContainer" containerID="397376842bf6deee1e38fadd7542309070a46d6c7496158c7df3b26c92172c7b" Nov 02 18:30:16.970307 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:16.970271 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4_odh-ai-gateway-infra(ac238df9-b947-47d0-a52a-38145c23f61a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" podUID="ac238df9-b947-47d0-a52a-38145c23f61a" Nov 02 18:30:16.979014 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:16.978980 5601 scope.go:122] "RemoveContainer" containerID="aa37258f1e1d1d74e8509017ddc82a8de7cb2206a7321f448bbb9af211e14627" Nov 02 18:30:17.152913 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:17.152880 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" Nov 02 18:30:17.270059 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:17.269977 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d5a3bb4-a17b-41cf-b512-f4de4bd0e471-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5a3bb4-a17b-41cf-b512-f4de4bd0e471-maas-api-service-ca\") pod \"1d5a3bb4-a17b-41cf-b512-f4de4bd0e471\" (UID: \"1d5a3bb4-a17b-41cf-b512-f4de4bd0e471\") " Nov 02 18:30:17.270224 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:17.270178 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d5a3bb4-a17b-41cf-b512-f4de4bd0e471-kube-api-access-ql9pj\" (UniqueName: \"kubernetes.io/projected/1d5a3bb4-a17b-41cf-b512-f4de4bd0e471-kube-api-access-ql9pj\") pod \"1d5a3bb4-a17b-41cf-b512-f4de4bd0e471\" (UID: \"1d5a3bb4-a17b-41cf-b512-f4de4bd0e471\") " Nov 02 18:30:17.270417 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:17.270393 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5a3bb4-a17b-41cf-b512-f4de4bd0e471-maas-api-service-ca" pod "1d5a3bb4-a17b-41cf-b512-f4de4bd0e471" (UID: "1d5a3bb4-a17b-41cf-b512-f4de4bd0e471"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:17.272369 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:17.272345 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5a3bb4-a17b-41cf-b512-f4de4bd0e471-kube-api-access-ql9pj" pod "1d5a3bb4-a17b-41cf-b512-f4de4bd0e471" (UID: "1d5a3bb4-a17b-41cf-b512-f4de4bd0e471"). InnerVolumeSpecName "kube-api-access-ql9pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:17.371145 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:17.371110 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql9pj\" (UniqueName: \"kubernetes.io/projected/1d5a3bb4-a17b-41cf-b512-f4de4bd0e471-kube-api-access-ql9pj\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:17.371145 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:17.371137 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5a3bb4-a17b-41cf-b512-f4de4bd0e471-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:17.419645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:17.419585 5601 scope.go:122] "RemoveContainer" containerID="dbfa2073ccb506ce0485749b9a7afbca754e72727b78ba2bb54657e6d2f7201d" Nov 02 18:30:17.429597 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:17.429568 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" path="/var/lib/kubelet/pods/ca8be970-ee6a-4fbe-94b1-c63f75fa4fda/volumes" Nov 02 18:30:17.946063 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:17.945980 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" podUID="28611469-b9c9-435a-87e1-1b7b37f0c5ff" containerName="payload-processing" containerID="cri-o://5c583e5433b1673b7c0df28429c0344587a791275595b4d4588e57ac24ddfd08" gracePeriod=25 Nov 02 18:30:17.980705 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:17.980675 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" Nov 02 18:30:17.981189 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:17.980707 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9" event={"ID":"1d5a3bb4-a17b-41cf-b512-f4de4bd0e471","Type":"ContainerDied","Data":"cbd10501bb9b3ad60f6638275848ac40ac5a0214cc3066c74d5037856297a535"} Nov 02 18:30:17.981189 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:17.980765 5601 scope.go:122] "RemoveContainer" containerID="57aab209da16b636d7c955bd00fc2e9a1d05c70a4be453cfd96e05a0b0c406d2" Nov 02 18:30:17.985887 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:17.985846 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" event={"ID":"4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99","Type":"ContainerStarted","Data":"6788c424c19215ca910b53ee90fe541e243024dd6b11ed6a6c60cdf1bfe9a567"} Nov 02 18:30:17.992886 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:17.992865 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q"] Nov 02 18:30:18.003206 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.003181 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9"] Nov 02 18:30:18.015873 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.015845 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894070-7dck9"] Nov 02 18:30:18.021104 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.021070 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4"] Nov 02 18:30:18.043160 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.043111 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" podStartSLOduration=18.043095501 podStartE2EDuration="18.043095501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:18.036989749 +0000 UTC m=+3382.728043762" watchObservedRunningTime="2026-11-02 18:30:18.043095501 +0000 UTC m=+3382.734149543" Nov 02 18:30:18.060172 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.060132 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" podUID="28611469-b9c9-435a-87e1-1b7b37f0c5ff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:30:18.210068 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.209988 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" Nov 02 18:30:18.254572 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.254537 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" Nov 02 18:30:18.283005 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.282974 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb3a82f-14e6-4cdf-8fa7-12db317f98f8-kube-api-access-plwx5\" (UniqueName: \"kubernetes.io/projected/deb3a82f-14e6-4cdf-8fa7-12db317f98f8-kube-api-access-plwx5\") pod \"deb3a82f-14e6-4cdf-8fa7-12db317f98f8\" (UID: \"deb3a82f-14e6-4cdf-8fa7-12db317f98f8\") " Nov 02 18:30:18.283187 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.283119 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deb3a82f-14e6-4cdf-8fa7-12db317f98f8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb3a82f-14e6-4cdf-8fa7-12db317f98f8-maas-api-service-ca\") pod \"deb3a82f-14e6-4cdf-8fa7-12db317f98f8\" (UID: \"deb3a82f-14e6-4cdf-8fa7-12db317f98f8\") " Nov 02 18:30:18.286903 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.285538 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb3a82f-14e6-4cdf-8fa7-12db317f98f8-maas-api-service-ca" pod "deb3a82f-14e6-4cdf-8fa7-12db317f98f8" (UID: "deb3a82f-14e6-4cdf-8fa7-12db317f98f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:18.286903 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.286114 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb3a82f-14e6-4cdf-8fa7-12db317f98f8-kube-api-access-plwx5" pod "deb3a82f-14e6-4cdf-8fa7-12db317f98f8" (UID: "deb3a82f-14e6-4cdf-8fa7-12db317f98f8"). InnerVolumeSpecName "kube-api-access-plwx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:18.384210 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.384179 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac238df9-b947-47d0-a52a-38145c23f61a-kube-api-access-8d8tx\" (UniqueName: \"kubernetes.io/projected/ac238df9-b947-47d0-a52a-38145c23f61a-kube-api-access-8d8tx\") pod \"ac238df9-b947-47d0-a52a-38145c23f61a\" (UID: \"ac238df9-b947-47d0-a52a-38145c23f61a\") " Nov 02 18:30:18.384410 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.384291 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac238df9-b947-47d0-a52a-38145c23f61a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac238df9-b947-47d0-a52a-38145c23f61a-maas-api-service-ca\") pod \"ac238df9-b947-47d0-a52a-38145c23f61a\" (UID: \"ac238df9-b947-47d0-a52a-38145c23f61a\") " Nov 02 18:30:18.384772 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.384749 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb3a82f-14e6-4cdf-8fa7-12db317f98f8-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:18.384772 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.384771 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plwx5\" (UniqueName: \"kubernetes.io/projected/deb3a82f-14e6-4cdf-8fa7-12db317f98f8-kube-api-access-plwx5\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:18.384975 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.384869 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac238df9-b947-47d0-a52a-38145c23f61a-maas-api-service-ca" pod "ac238df9-b947-47d0-a52a-38145c23f61a" (UID: "ac238df9-b947-47d0-a52a-38145c23f61a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:18.386401 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.386369 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac238df9-b947-47d0-a52a-38145c23f61a-kube-api-access-8d8tx" pod "ac238df9-b947-47d0-a52a-38145c23f61a" (UID: "ac238df9-b947-47d0-a52a-38145c23f61a"). InnerVolumeSpecName "kube-api-access-8d8tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:18.419150 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.419125 5601 scope.go:122] "RemoveContainer" containerID="52bfd3200ed0bb00952d9d2a3b7fb9e4968b0eb7abeac928dead17380230f2f7" Nov 02 18:30:18.486291 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.486210 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d8tx\" (UniqueName: \"kubernetes.io/projected/ac238df9-b947-47d0-a52a-38145c23f61a-kube-api-access-8d8tx\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:18.486291 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:18.486257 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac238df9-b947-47d0-a52a-38145c23f61a-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.000714 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.000624 5601 generic.go:356] "Generic (PLEG): container finished" podID="4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99" containerID="6788c424c19215ca910b53ee90fe541e243024dd6b11ed6a6c60cdf1bfe9a567" exitCode=60 Nov 02 18:30:19.001171 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.000708 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" event={"ID":"4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99","Type":"ContainerDied","Data":"6788c424c19215ca910b53ee90fe541e243024dd6b11ed6a6c60cdf1bfe9a567"} Nov 02 18:30:19.001171 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.000786 5601 scope.go:122] "RemoveContainer" containerID="dbfa2073ccb506ce0485749b9a7afbca754e72727b78ba2bb54657e6d2f7201d" Nov 02 18:30:19.001358 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.001186 5601 scope.go:122] "RemoveContainer" containerID="6788c424c19215ca910b53ee90fe541e243024dd6b11ed6a6c60cdf1bfe9a567" Nov 02 18:30:19.001630 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:19.001469 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6_odh-ai-gateway-infra(4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" podUID="4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99" Nov 02 18:30:19.008886 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.008842 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" event={"ID":"01186ed8-9a15-4802-924a-85dd46e7dc8b","Type":"ContainerStarted","Data":"7e750bfd0684273f207b687b0d0d6a00e8b1039f15954e1efa7f15e0772c69e0"} Nov 02 18:30:19.013782 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.013754 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" event={"ID":"deb3a82f-14e6-4cdf-8fa7-12db317f98f8","Type":"ContainerDied","Data":"d26fa7f94ce1717f1e47457bf0101484e0e83e32fff777ba9f1a048d2e5eb2fc"} Nov 02 18:30:19.013908 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.013794 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q" Nov 02 18:30:19.017563 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.017442 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" event={"ID":"ac238df9-b947-47d0-a52a-38145c23f61a","Type":"ContainerDied","Data":"ff716db9d2ad8311d8bb53b0dd897dd414307f8716fe675cafaa9bad68312f85"} Nov 02 18:30:19.017563 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.017556 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4" Nov 02 18:30:19.024963 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.024940 5601 scope.go:122] "RemoveContainer" containerID="23aefc63bd420d65c4d4edce208881567fb36c8924593d9a8415fc63d78fceb1" Nov 02 18:30:19.084804 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.077661 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6"] Nov 02 18:30:19.091397 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.090099 5601 scope.go:122] "RemoveContainer" containerID="397376842bf6deee1e38fadd7542309070a46d6c7496158c7df3b26c92172c7b" Nov 02 18:30:19.118297 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.118268 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4"] Nov 02 18:30:19.124023 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.124003 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894070-9hpj4"] Nov 02 18:30:19.127871 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.127729 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q"] Nov 02 18:30:19.132231 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.132207 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894070-g2r6q"] Nov 02 18:30:19.420696 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.420654 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" containerID="cri-o://ca1c8f2ce9f27230e354ac817f1246aacb41e71b53b4b8efd62ddf21d94d1c93" gracePeriod=45 Nov 02 18:30:19.426749 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.426727 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d5a3bb4-a17b-41cf-b512-f4de4bd0e471" path="/var/lib/kubelet/pods/1d5a3bb4-a17b-41cf-b512-f4de4bd0e471/volumes" Nov 02 18:30:19.427149 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.427136 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac238df9-b947-47d0-a52a-38145c23f61a" path="/var/lib/kubelet/pods/ac238df9-b947-47d0-a52a-38145c23f61a/volumes" Nov 02 18:30:19.427968 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.427946 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deb3a82f-14e6-4cdf-8fa7-12db317f98f8" path="/var/lib/kubelet/pods/deb3a82f-14e6-4cdf-8fa7-12db317f98f8/volumes" Nov 02 18:30:19.695974 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.695946 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 18:30:19.820152 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.820121 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c749700-fae4-498d-9fb1-38652eec3eef-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c749700-fae4-498d-9fb1-38652eec3eef-tls-certs\") pod \"5c749700-fae4-498d-9fb1-38652eec3eef\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " Nov 02 18:30:19.820801 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.820773 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-model-cache\") pod \"5c749700-fae4-498d-9fb1-38652eec3eef\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " Nov 02 18:30:19.820946 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.820841 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-dshm\") pod \"5c749700-fae4-498d-9fb1-38652eec3eef\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " Nov 02 18:30:19.821011 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.820923 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-home\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-home\") pod \"5c749700-fae4-498d-9fb1-38652eec3eef\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " Nov 02 18:30:19.821011 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.821005 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c749700-fae4-498d-9fb1-38652eec3eef-kube-api-access-hp254\" (UniqueName: \"kubernetes.io/projected/5c749700-fae4-498d-9fb1-38652eec3eef-kube-api-access-hp254\") pod \"5c749700-fae4-498d-9fb1-38652eec3eef\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " Nov 02 18:30:19.821141 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.821005 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-model-cache" pod "5c749700-fae4-498d-9fb1-38652eec3eef" (UID: "5c749700-fae4-498d-9fb1-38652eec3eef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:19.821141 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.821104 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-tmp-dir\") pod \"5c749700-fae4-498d-9fb1-38652eec3eef\" (UID: \"5c749700-fae4-498d-9fb1-38652eec3eef\") " Nov 02 18:30:19.821835 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.821795 5601 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-model-cache\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.822027 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.821990 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-tmp-dir" pod "5c749700-fae4-498d-9fb1-38652eec3eef" (UID: "5c749700-fae4-498d-9fb1-38652eec3eef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:19.822510 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.822483 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-home" pod "5c749700-fae4-498d-9fb1-38652eec3eef" (UID: "5c749700-fae4-498d-9fb1-38652eec3eef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:19.824231 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.824203 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-dshm" pod "5c749700-fae4-498d-9fb1-38652eec3eef" (UID: "5c749700-fae4-498d-9fb1-38652eec3eef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:19.827076 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.827035 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c749700-fae4-498d-9fb1-38652eec3eef-tls-certs" pod "5c749700-fae4-498d-9fb1-38652eec3eef" (UID: "5c749700-fae4-498d-9fb1-38652eec3eef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:30:19.831088 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.831059 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c749700-fae4-498d-9fb1-38652eec3eef-kube-api-access-hp254" pod "5c749700-fae4-498d-9fb1-38652eec3eef" (UID: "5c749700-fae4-498d-9fb1-38652eec3eef"). InnerVolumeSpecName "kube-api-access-hp254". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:19.924976 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.924936 5601 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-dshm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.924976 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.924975 5601 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-home\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.925378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.924989 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp254\" (UniqueName: \"kubernetes.io/projected/5c749700-fae4-498d-9fb1-38652eec3eef-kube-api-access-hp254\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.925378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.925003 5601 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c749700-fae4-498d-9fb1-38652eec3eef-tmp-dir\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.925378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:19.925018 5601 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c749700-fae4-498d-9fb1-38652eec3eef-tls-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:20.026611 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.026576 5601 generic.go:356] "Generic (PLEG): container finished" podID="5c749700-fae4-498d-9fb1-38652eec3eef" containerID="ca1c8f2ce9f27230e354ac817f1246aacb41e71b53b4b8efd62ddf21d94d1c93" exitCode=0 Nov 02 18:30:20.027096 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.026657 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" Nov 02 18:30:20.027096 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.026652 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" event={"ID":"5c749700-fae4-498d-9fb1-38652eec3eef","Type":"ContainerDied","Data":"ca1c8f2ce9f27230e354ac817f1246aacb41e71b53b4b8efd62ddf21d94d1c93"} Nov 02 18:30:20.027096 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.026791 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6" event={"ID":"5c749700-fae4-498d-9fb1-38652eec3eef","Type":"ContainerDied","Data":"28647a607b76d0e31c6dea82099c4e0b2270c28e16a1d5ec1601c551e8294dd7"} Nov 02 18:30:20.027096 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.026821 5601 scope.go:122] "RemoveContainer" containerID="ca1c8f2ce9f27230e354ac817f1246aacb41e71b53b4b8efd62ddf21d94d1c93" Nov 02 18:30:20.029539 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.029516 5601 generic.go:356] "Generic (PLEG): container finished" podID="01186ed8-9a15-4802-924a-85dd46e7dc8b" containerID="7e750bfd0684273f207b687b0d0d6a00e8b1039f15954e1efa7f15e0772c69e0" exitCode=60 Nov 02 18:30:20.029663 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.029579 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" event={"ID":"01186ed8-9a15-4802-924a-85dd46e7dc8b","Type":"ContainerDied","Data":"7e750bfd0684273f207b687b0d0d6a00e8b1039f15954e1efa7f15e0772c69e0"} Nov 02 18:30:20.030050 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.030029 5601 scope.go:122] "RemoveContainer" containerID="7e750bfd0684273f207b687b0d0d6a00e8b1039f15954e1efa7f15e0772c69e0" Nov 02 18:30:20.031266 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:20.030330 5601 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms_odh-ai-gateway-infra(01186ed8-9a15-4802-924a-85dd46e7dc8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" podUID="01186ed8-9a15-4802-924a-85dd46e7dc8b" Nov 02 18:30:20.040040 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.040012 5601 scope.go:122] "RemoveContainer" containerID="ca1c8f2ce9f27230e354ac817f1246aacb41e71b53b4b8efd62ddf21d94d1c93" Nov 02 18:30:20.040312 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:20.040289 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca1c8f2ce9f27230e354ac817f1246aacb41e71b53b4b8efd62ddf21d94d1c93\": container with ID starting with ca1c8f2ce9f27230e354ac817f1246aacb41e71b53b4b8efd62ddf21d94d1c93 not found: ID does not exist" containerID="ca1c8f2ce9f27230e354ac817f1246aacb41e71b53b4b8efd62ddf21d94d1c93" Nov 02 18:30:20.040389 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.040326 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca1c8f2ce9f27230e354ac817f1246aacb41e71b53b4b8efd62ddf21d94d1c93"} err="failed to get container status \"ca1c8f2ce9f27230e354ac817f1246aacb41e71b53b4b8efd62ddf21d94d1c93\": rpc error: code = NotFound desc = could not find container \"ca1c8f2ce9f27230e354ac817f1246aacb41e71b53b4b8efd62ddf21d94d1c93\": container with ID starting with ca1c8f2ce9f27230e354ac817f1246aacb41e71b53b4b8efd62ddf21d94d1c93 not found: ID does not exist" Nov 02 18:30:20.040389 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.040350 5601 scope.go:122] "RemoveContainer" containerID="52bfd3200ed0bb00952d9d2a3b7fb9e4968b0eb7abeac928dead17380230f2f7" Nov 02 18:30:20.055689 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.055659 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms"] Nov 02 18:30:20.083012 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.082578 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6"] Nov 02 18:30:20.086199 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.086168 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/premium-w3-a37cf0-kserve-8d64b668c-96tj6"] Nov 02 18:30:20.187049 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.186976 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" Nov 02 18:30:20.329850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.329814 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99-maas-api-service-ca\") pod \"4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99\" (UID: \"4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99\") " Nov 02 18:30:20.330029 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.329924 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99-kube-api-access-dfdvw\" (UniqueName: \"kubernetes.io/projected/4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99-kube-api-access-dfdvw\") pod \"4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99\" (UID: \"4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99\") " Nov 02 18:30:20.330199 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.330176 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99-maas-api-service-ca" pod "4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99" (UID: "4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:20.332131 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.332091 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99-kube-api-access-dfdvw" pod "4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99" (UID: "4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99"). InnerVolumeSpecName "kube-api-access-dfdvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:20.430815 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.430785 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:20.430815 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:20.430810 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfdvw\" (UniqueName: \"kubernetes.io/projected/4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99-kube-api-access-dfdvw\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:21.042896 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:21.042817 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" Nov 02 18:30:21.042896 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:21.042849 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6" event={"ID":"4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99","Type":"ContainerDied","Data":"96082e492c1c1374988ede5a5a4c7dbadd6b78e4d8e4d7d6c78aa5383482d69b"} Nov 02 18:30:21.043382 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:21.042910 5601 scope.go:122] "RemoveContainer" containerID="6788c424c19215ca910b53ee90fe541e243024dd6b11ed6a6c60cdf1bfe9a567" Nov 02 18:30:21.079788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:21.079752 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6"] Nov 02 18:30:21.082247 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:21.082223 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894070-z9wx6"] Nov 02 18:30:21.200756 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:21.200728 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" Nov 02 18:30:21.343050 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:21.342968 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01186ed8-9a15-4802-924a-85dd46e7dc8b-kube-api-access-znzrt\" (UniqueName: \"kubernetes.io/projected/01186ed8-9a15-4802-924a-85dd46e7dc8b-kube-api-access-znzrt\") pod \"01186ed8-9a15-4802-924a-85dd46e7dc8b\" (UID: \"01186ed8-9a15-4802-924a-85dd46e7dc8b\") " Nov 02 18:30:21.343205 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:21.343154 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01186ed8-9a15-4802-924a-85dd46e7dc8b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01186ed8-9a15-4802-924a-85dd46e7dc8b-maas-api-service-ca\") pod \"01186ed8-9a15-4802-924a-85dd46e7dc8b\" (UID: \"01186ed8-9a15-4802-924a-85dd46e7dc8b\") " Nov 02 18:30:21.343539 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:21.343500 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01186ed8-9a15-4802-924a-85dd46e7dc8b-maas-api-service-ca" pod "01186ed8-9a15-4802-924a-85dd46e7dc8b" (UID: "01186ed8-9a15-4802-924a-85dd46e7dc8b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:21.343710 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:21.343693 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01186ed8-9a15-4802-924a-85dd46e7dc8b-maas-api-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:21.345142 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:21.345114 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01186ed8-9a15-4802-924a-85dd46e7dc8b-kube-api-access-znzrt" pod "01186ed8-9a15-4802-924a-85dd46e7dc8b" (UID: "01186ed8-9a15-4802-924a-85dd46e7dc8b"). InnerVolumeSpecName "kube-api-access-znzrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:21.427598 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:21.427559 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99" path="/var/lib/kubelet/pods/4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99/volumes" Nov 02 18:30:21.428112 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:21.428089 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" path="/var/lib/kubelet/pods/5c749700-fae4-498d-9fb1-38652eec3eef/volumes" Nov 02 18:30:21.448834 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:21.446237 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znzrt\" (UniqueName: \"kubernetes.io/projected/01186ed8-9a15-4802-924a-85dd46e7dc8b-kube-api-access-znzrt\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:22.053185 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:22.053154 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" Nov 02 18:30:22.053711 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:22.053188 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms" event={"ID":"01186ed8-9a15-4802-924a-85dd46e7dc8b","Type":"ContainerDied","Data":"22a431637de0d25e0f4fed359566e08947feb25ac8227fe88278ea9f1ca11ca8"} Nov 02 18:30:22.053711 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:22.053291 5601 scope.go:122] "RemoveContainer" containerID="7e750bfd0684273f207b687b0d0d6a00e8b1039f15954e1efa7f15e0772c69e0" Nov 02 18:30:22.071813 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:22.071786 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms"] Nov 02 18:30:22.076775 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:22.076750 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-36a991-29894070-94mms"] Nov 02 18:30:23.426158 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:23.426127 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01186ed8-9a15-4802-924a-85dd46e7dc8b" path="/var/lib/kubelet/pods/01186ed8-9a15-4802-924a-85dd46e7dc8b/volumes" Nov 02 18:30:28.058542 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:28.058505 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" podUID="28611469-b9c9-435a-87e1-1b7b37f0c5ff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:30:29.110565 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:29.110527 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p"] Nov 02 18:30:29.111129 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:29.110795 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" podUID="abc3edc9-ba31-479b-97f7-5eee71b5301b" containerName="maas-api" containerID="cri-o://7a50bde6181623e4669f49babdb8a29d88b3133266fa0b2f92099f067188c88a" gracePeriod=30 Nov 02 18:30:29.545117 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:29.545088 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" Nov 02 18:30:29.584331 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:29.584295 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc3edc9-ba31-479b-97f7-5eee71b5301b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abc3edc9-ba31-479b-97f7-5eee71b5301b-maas-api-tls\") pod \"abc3edc9-ba31-479b-97f7-5eee71b5301b\" (UID: \"abc3edc9-ba31-479b-97f7-5eee71b5301b\") " Nov 02 18:30:29.584501 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:29.584392 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc3edc9-ba31-479b-97f7-5eee71b5301b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc3edc9-ba31-479b-97f7-5eee71b5301b-openshift-service-ca\") pod \"abc3edc9-ba31-479b-97f7-5eee71b5301b\" (UID: \"abc3edc9-ba31-479b-97f7-5eee71b5301b\") " Nov 02 18:30:29.584592 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:29.584541 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc3edc9-ba31-479b-97f7-5eee71b5301b-kube-api-access-gvkfz\" (UniqueName: \"kubernetes.io/projected/abc3edc9-ba31-479b-97f7-5eee71b5301b-kube-api-access-gvkfz\") pod \"abc3edc9-ba31-479b-97f7-5eee71b5301b\" (UID: \"abc3edc9-ba31-479b-97f7-5eee71b5301b\") " Nov 02 18:30:29.584690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:29.584656 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc3edc9-ba31-479b-97f7-5eee71b5301b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/abc3edc9-ba31-479b-97f7-5eee71b5301b-metrics-certs\") pod \"abc3edc9-ba31-479b-97f7-5eee71b5301b\" (UID: \"abc3edc9-ba31-479b-97f7-5eee71b5301b\") " Nov 02 18:30:29.584826 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:29.584798 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc3edc9-ba31-479b-97f7-5eee71b5301b-openshift-service-ca" pod "abc3edc9-ba31-479b-97f7-5eee71b5301b" (UID: "abc3edc9-ba31-479b-97f7-5eee71b5301b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:29.585152 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:29.585127 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc3edc9-ba31-479b-97f7-5eee71b5301b-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:29.586750 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:29.586723 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc3edc9-ba31-479b-97f7-5eee71b5301b-maas-api-tls" pod "abc3edc9-ba31-479b-97f7-5eee71b5301b" (UID: "abc3edc9-ba31-479b-97f7-5eee71b5301b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:30:29.587396 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:29.587367 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc3edc9-ba31-479b-97f7-5eee71b5301b-metrics-certs" pod "abc3edc9-ba31-479b-97f7-5eee71b5301b" (UID: "abc3edc9-ba31-479b-97f7-5eee71b5301b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:30:29.587557 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:29.587537 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc3edc9-ba31-479b-97f7-5eee71b5301b-kube-api-access-gvkfz" pod "abc3edc9-ba31-479b-97f7-5eee71b5301b" (UID: "abc3edc9-ba31-479b-97f7-5eee71b5301b"). InnerVolumeSpecName "kube-api-access-gvkfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:29.686367 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:29.686325 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abc3edc9-ba31-479b-97f7-5eee71b5301b-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:29.686367 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:29.686368 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abc3edc9-ba31-479b-97f7-5eee71b5301b-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:29.686585 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:29.686384 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvkfz\" (UniqueName: \"kubernetes.io/projected/abc3edc9-ba31-479b-97f7-5eee71b5301b-kube-api-access-gvkfz\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:30.116271 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:30.116234 5601 generic.go:356] "Generic (PLEG): container finished" podID="abc3edc9-ba31-479b-97f7-5eee71b5301b" containerID="7a50bde6181623e4669f49babdb8a29d88b3133266fa0b2f92099f067188c88a" exitCode=0 Nov 02 18:30:30.116759 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:30.116269 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" event={"ID":"abc3edc9-ba31-479b-97f7-5eee71b5301b","Type":"ContainerDied","Data":"7a50bde6181623e4669f49babdb8a29d88b3133266fa0b2f92099f067188c88a"} Nov 02 18:30:30.116759 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:30.116334 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" event={"ID":"abc3edc9-ba31-479b-97f7-5eee71b5301b","Type":"ContainerDied","Data":"8193a8acee95e0d75121d240380edc178d26af838b309f02189df160895a998c"} Nov 02 18:30:30.116759 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:30.116357 5601 scope.go:122] "RemoveContainer" containerID="7a50bde6181623e4669f49babdb8a29d88b3133266fa0b2f92099f067188c88a" Nov 02 18:30:30.116759 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:30.116286 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p" Nov 02 18:30:30.132025 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:30.132004 5601 scope.go:122] "RemoveContainer" containerID="7a50bde6181623e4669f49babdb8a29d88b3133266fa0b2f92099f067188c88a" Nov 02 18:30:30.132336 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:30.132314 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a50bde6181623e4669f49babdb8a29d88b3133266fa0b2f92099f067188c88a\": container with ID starting with 7a50bde6181623e4669f49babdb8a29d88b3133266fa0b2f92099f067188c88a not found: ID does not exist" containerID="7a50bde6181623e4669f49babdb8a29d88b3133266fa0b2f92099f067188c88a" Nov 02 18:30:30.132397 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:30.132347 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a50bde6181623e4669f49babdb8a29d88b3133266fa0b2f92099f067188c88a"} err="failed to get container status \"7a50bde6181623e4669f49babdb8a29d88b3133266fa0b2f92099f067188c88a\": rpc error: code = NotFound desc = could not find container \"7a50bde6181623e4669f49babdb8a29d88b3133266fa0b2f92099f067188c88a\": container with ID starting with 7a50bde6181623e4669f49babdb8a29d88b3133266fa0b2f92099f067188c88a not found: ID does not exist" Nov 02 18:30:30.146658 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:30.146634 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p"] Nov 02 18:30:30.151068 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:30.151045 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b1a788-7589dc6b56-vsq7p"] Nov 02 18:30:30.657046 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:30.654956 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677"] Nov 02 18:30:30.657046 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:30.655365 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" podUID="1b0271e7-cb07-45b2-a349-32f49ce6e442" containerName="maas-api" containerID="cri-o://70f34271d75fae001ba2b1111b63b47cd40a3f107fb97483e0961a40c238e597" gracePeriod=30 Nov 02 18:30:31.070277 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.070251 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" Nov 02 18:30:31.105873 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.105840 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b0271e7-cb07-45b2-a349-32f49ce6e442-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b0271e7-cb07-45b2-a349-32f49ce6e442-maas-api-tls\") pod \"1b0271e7-cb07-45b2-a349-32f49ce6e442\" (UID: \"1b0271e7-cb07-45b2-a349-32f49ce6e442\") " Nov 02 18:30:31.105873 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.105875 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b0271e7-cb07-45b2-a349-32f49ce6e442-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0271e7-cb07-45b2-a349-32f49ce6e442-openshift-service-ca\") pod \"1b0271e7-cb07-45b2-a349-32f49ce6e442\" (UID: \"1b0271e7-cb07-45b2-a349-32f49ce6e442\") " Nov 02 18:30:31.106113 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.105918 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b0271e7-cb07-45b2-a349-32f49ce6e442-kube-api-access-qx42v\" (UniqueName: \"kubernetes.io/projected/1b0271e7-cb07-45b2-a349-32f49ce6e442-kube-api-access-qx42v\") pod \"1b0271e7-cb07-45b2-a349-32f49ce6e442\" (UID: \"1b0271e7-cb07-45b2-a349-32f49ce6e442\") " Nov 02 18:30:31.106113 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.105934 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b0271e7-cb07-45b2-a349-32f49ce6e442-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b0271e7-cb07-45b2-a349-32f49ce6e442-metrics-certs\") pod \"1b0271e7-cb07-45b2-a349-32f49ce6e442\" (UID: \"1b0271e7-cb07-45b2-a349-32f49ce6e442\") " Nov 02 18:30:31.106353 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.106323 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0271e7-cb07-45b2-a349-32f49ce6e442-openshift-service-ca" pod "1b0271e7-cb07-45b2-a349-32f49ce6e442" (UID: "1b0271e7-cb07-45b2-a349-32f49ce6e442"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:31.106461 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.106442 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0271e7-cb07-45b2-a349-32f49ce6e442-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:31.108159 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.108076 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b0271e7-cb07-45b2-a349-32f49ce6e442-metrics-certs" pod "1b0271e7-cb07-45b2-a349-32f49ce6e442" (UID: "1b0271e7-cb07-45b2-a349-32f49ce6e442"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:30:31.108508 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.108478 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b0271e7-cb07-45b2-a349-32f49ce6e442-maas-api-tls" pod "1b0271e7-cb07-45b2-a349-32f49ce6e442" (UID: "1b0271e7-cb07-45b2-a349-32f49ce6e442"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:30:31.108667 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.108579 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0271e7-cb07-45b2-a349-32f49ce6e442-kube-api-access-qx42v" pod "1b0271e7-cb07-45b2-a349-32f49ce6e442" (UID: "1b0271e7-cb07-45b2-a349-32f49ce6e442"). InnerVolumeSpecName "kube-api-access-qx42v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:31.130171 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.130142 5601 generic.go:356] "Generic (PLEG): container finished" podID="1b0271e7-cb07-45b2-a349-32f49ce6e442" containerID="70f34271d75fae001ba2b1111b63b47cd40a3f107fb97483e0961a40c238e597" exitCode=0 Nov 02 18:30:31.130583 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.130187 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" event={"ID":"1b0271e7-cb07-45b2-a349-32f49ce6e442","Type":"ContainerDied","Data":"70f34271d75fae001ba2b1111b63b47cd40a3f107fb97483e0961a40c238e597"} Nov 02 18:30:31.130583 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.130207 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" Nov 02 18:30:31.130583 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.130243 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677" event={"ID":"1b0271e7-cb07-45b2-a349-32f49ce6e442","Type":"ContainerDied","Data":"e24c0ac9637e8cd3d6f90483b207baecaba270e40089a291311ea0e86b000679"} Nov 02 18:30:31.130583 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.130275 5601 scope.go:122] "RemoveContainer" containerID="70f34271d75fae001ba2b1111b63b47cd40a3f107fb97483e0961a40c238e597" Nov 02 18:30:31.142745 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.142720 5601 scope.go:122] "RemoveContainer" containerID="70f34271d75fae001ba2b1111b63b47cd40a3f107fb97483e0961a40c238e597" Nov 02 18:30:31.143056 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:31.143032 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f34271d75fae001ba2b1111b63b47cd40a3f107fb97483e0961a40c238e597\": container with ID starting with 70f34271d75fae001ba2b1111b63b47cd40a3f107fb97483e0961a40c238e597 not found: ID does not exist" containerID="70f34271d75fae001ba2b1111b63b47cd40a3f107fb97483e0961a40c238e597" Nov 02 18:30:31.143132 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.143066 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f34271d75fae001ba2b1111b63b47cd40a3f107fb97483e0961a40c238e597"} err="failed to get container status \"70f34271d75fae001ba2b1111b63b47cd40a3f107fb97483e0961a40c238e597\": rpc error: code = NotFound desc = could not find container \"70f34271d75fae001ba2b1111b63b47cd40a3f107fb97483e0961a40c238e597\": container with ID starting with 70f34271d75fae001ba2b1111b63b47cd40a3f107fb97483e0961a40c238e597 not found: ID does not exist" Nov 02 18:30:31.156817 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.156796 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677"] Nov 02 18:30:31.162659 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.162639 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a37cf0-6bdfc8fb4-sr677"] Nov 02 18:30:31.211670 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.209921 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b0271e7-cb07-45b2-a349-32f49ce6e442-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:31.211670 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.209967 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx42v\" (UniqueName: \"kubernetes.io/projected/1b0271e7-cb07-45b2-a349-32f49ce6e442-kube-api-access-qx42v\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:31.211670 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.209987 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b0271e7-cb07-45b2-a349-32f49ce6e442-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:31.426352 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.426317 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b0271e7-cb07-45b2-a349-32f49ce6e442" path="/var/lib/kubelet/pods/1b0271e7-cb07-45b2-a349-32f49ce6e442/volumes" Nov 02 18:30:31.426790 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:31.426774 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc3edc9-ba31-479b-97f7-5eee71b5301b" path="/var/lib/kubelet/pods/abc3edc9-ba31-479b-97f7-5eee71b5301b/volumes" Nov 02 18:30:33.539638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:33.538906 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw"] Nov 02 18:30:33.539638 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:33.539162 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" podUID="ee0d3ab7-7076-4998-b997-2fa88e7622ad" containerName="istio-proxy" containerID="cri-o://4cdec65986f89463578aabf9ff0a1b1f818b048b74925193184dc1a3f97eef06" gracePeriod=30 Nov 02 18:30:34.484863 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:34.484818 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:30:38.057633 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:30:38.057598 5601 logging.go:55] [core] [Channel #3196 SubChannel #3197]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.52:50052", ServerName: "10.134.0.52:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.52:50052: connect: connection refused" Nov 02 18:30:38.185226 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.185194 5601 generic.go:356] "Generic (PLEG): container finished" podID="28611469-b9c9-435a-87e1-1b7b37f0c5ff" containerID="5c583e5433b1673b7c0df28429c0344587a791275595b4d4588e57ac24ddfd08" exitCode=0 Nov 02 18:30:38.185377 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.185239 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" event={"ID":"28611469-b9c9-435a-87e1-1b7b37f0c5ff","Type":"ContainerDied","Data":"5c583e5433b1673b7c0df28429c0344587a791275595b4d4588e57ac24ddfd08"} Nov 02 18:30:38.242945 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.242920 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" Nov 02 18:30:38.318220 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.318189 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28611469-b9c9-435a-87e1-1b7b37f0c5ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28611469-b9c9-435a-87e1-1b7b37f0c5ff-plugins-config-volume\") pod \"28611469-b9c9-435a-87e1-1b7b37f0c5ff\" (UID: \"28611469-b9c9-435a-87e1-1b7b37f0c5ff\") " Nov 02 18:30:38.318380 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.318242 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28611469-b9c9-435a-87e1-1b7b37f0c5ff-kube-api-access-4bdbq\" (UniqueName: \"kubernetes.io/projected/28611469-b9c9-435a-87e1-1b7b37f0c5ff-kube-api-access-4bdbq\") pod \"28611469-b9c9-435a-87e1-1b7b37f0c5ff\" (UID: \"28611469-b9c9-435a-87e1-1b7b37f0c5ff\") " Nov 02 18:30:38.318572 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.318546 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28611469-b9c9-435a-87e1-1b7b37f0c5ff-plugins-config-volume" pod "28611469-b9c9-435a-87e1-1b7b37f0c5ff" (UID: "28611469-b9c9-435a-87e1-1b7b37f0c5ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:38.320385 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.320364 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28611469-b9c9-435a-87e1-1b7b37f0c5ff-kube-api-access-4bdbq" pod "28611469-b9c9-435a-87e1-1b7b37f0c5ff" (UID: "28611469-b9c9-435a-87e1-1b7b37f0c5ff"). InnerVolumeSpecName "kube-api-access-4bdbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:38.419233 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.419198 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28611469-b9c9-435a-87e1-1b7b37f0c5ff-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:38.419233 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.419233 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bdbq\" (UniqueName: \"kubernetes.io/projected/28611469-b9c9-435a-87e1-1b7b37f0c5ff-kube-api-access-4bdbq\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:38.682058 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.682021 5601 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 18:30:38.682204 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.682100 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" podUID="ee0d3ab7-7076-4998-b997-2fa88e7622ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 18:30:38.826762 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.826718 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 18:30:38.925236 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.925203 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee0d3ab7-7076-4998-b997-2fa88e7622ad-kube-api-access-mffsw\" (UniqueName: \"kubernetes.io/projected/ee0d3ab7-7076-4998-b997-2fa88e7622ad-kube-api-access-mffsw\") pod \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " Nov 02 18:30:38.925427 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.925264 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-credential-socket\") pod \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " Nov 02 18:30:38.925427 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.925297 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istiod-ca-cert\") pod \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " Nov 02 18:30:38.925427 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.925363 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-podinfo\") pod \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " Nov 02 18:30:38.925427 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.925412 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-token\" (UniqueName: \"kubernetes.io/projected/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-token\") pod \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " Nov 02 18:30:38.925678 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.925445 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-workload-socket\") pod \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " Nov 02 18:30:38.925678 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.925555 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-workload-certs\") pod \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " Nov 02 18:30:38.925678 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.925608 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-credential-socket" pod "ee0d3ab7-7076-4998-b997-2fa88e7622ad" (UID: "ee0d3ab7-7076-4998-b997-2fa88e7622ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:38.925678 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.925592 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-data\") pod \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " Nov 02 18:30:38.925920 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.925732 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-envoy\") pod \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\" (UID: \"ee0d3ab7-7076-4998-b997-2fa88e7622ad\") " Nov 02 18:30:38.925920 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.925843 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-workload-certs" pod "ee0d3ab7-7076-4998-b997-2fa88e7622ad" (UID: "ee0d3ab7-7076-4998-b997-2fa88e7622ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:38.926047 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.925963 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istiod-ca-cert" pod "ee0d3ab7-7076-4998-b997-2fa88e7622ad" (UID: "ee0d3ab7-7076-4998-b997-2fa88e7622ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:38.926315 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.926282 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-data" pod "ee0d3ab7-7076-4998-b997-2fa88e7622ad" (UID: "ee0d3ab7-7076-4998-b997-2fa88e7622ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:38.926555 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.926528 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-workload-socket" pod "ee0d3ab7-7076-4998-b997-2fa88e7622ad" (UID: "ee0d3ab7-7076-4998-b997-2fa88e7622ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:38.926868 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.926843 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:38.926985 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.926873 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:38.926985 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.926887 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:38.926985 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.926901 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:38.926985 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.926916 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:38.928113 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.928087 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-podinfo" pod "ee0d3ab7-7076-4998-b997-2fa88e7622ad" (UID: "ee0d3ab7-7076-4998-b997-2fa88e7622ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:30:38.928285 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.928261 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0d3ab7-7076-4998-b997-2fa88e7622ad-kube-api-access-mffsw" pod "ee0d3ab7-7076-4998-b997-2fa88e7622ad" (UID: "ee0d3ab7-7076-4998-b997-2fa88e7622ad"). InnerVolumeSpecName "kube-api-access-mffsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:38.928515 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.928488 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-envoy" pod "ee0d3ab7-7076-4998-b997-2fa88e7622ad" (UID: "ee0d3ab7-7076-4998-b997-2fa88e7622ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:38.929091 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:38.929066 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-token" pod "ee0d3ab7-7076-4998-b997-2fa88e7622ad" (UID: "ee0d3ab7-7076-4998-b997-2fa88e7622ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:39.027668 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.027597 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:39.027668 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.027670 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:39.027920 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.027684 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0d3ab7-7076-4998-b997-2fa88e7622ad-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:39.027920 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.027695 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mffsw\" (UniqueName: \"kubernetes.io/projected/ee0d3ab7-7076-4998-b997-2fa88e7622ad-kube-api-access-mffsw\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:39.058262 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.058222 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" podUID="28611469-b9c9-435a-87e1-1b7b37f0c5ff" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.52:50052\" within 1s: context deadline exceeded" Nov 02 18:30:39.058741 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.058314 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" Nov 02 18:30:39.058741 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:30:39.058235 5601 logging.go:55] [core] [Channel #3196 SubChannel #3197]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.52:50052", ServerName: "10.134.0.52:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.52:50052: operation was canceled" Nov 02 18:30:39.193004 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.192975 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" Nov 02 18:30:39.193196 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.193019 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg" event={"ID":"28611469-b9c9-435a-87e1-1b7b37f0c5ff","Type":"ContainerDied","Data":"c7b686de7493d2b6388a1fd22a697511592e38b6710ac344b7312043e818bb64"} Nov 02 18:30:39.193196 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.193091 5601 scope.go:122] "RemoveContainer" containerID="5c583e5433b1673b7c0df28429c0344587a791275595b4d4588e57ac24ddfd08" Nov 02 18:30:39.194874 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.194846 5601 generic.go:356] "Generic (PLEG): container finished" podID="ee0d3ab7-7076-4998-b997-2fa88e7622ad" containerID="4cdec65986f89463578aabf9ff0a1b1f818b048b74925193184dc1a3f97eef06" exitCode=0 Nov 02 18:30:39.195019 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.194897 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" Nov 02 18:30:39.195019 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.194932 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" event={"ID":"ee0d3ab7-7076-4998-b997-2fa88e7622ad","Type":"ContainerDied","Data":"4cdec65986f89463578aabf9ff0a1b1f818b048b74925193184dc1a3f97eef06"} Nov 02 18:30:39.195019 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.194978 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw" event={"ID":"ee0d3ab7-7076-4998-b997-2fa88e7622ad","Type":"ContainerDied","Data":"0dd85333ed1b093c5cdf1d4ad4b1467011b41fffcd13690ee2c7c8c1a1ad07a4"} Nov 02 18:30:39.205187 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.205166 5601 scope.go:122] "RemoveContainer" containerID="4cdec65986f89463578aabf9ff0a1b1f818b048b74925193184dc1a3f97eef06" Nov 02 18:30:39.214289 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.214269 5601 scope.go:122] "RemoveContainer" containerID="4cdec65986f89463578aabf9ff0a1b1f818b048b74925193184dc1a3f97eef06" Nov 02 18:30:39.215178 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:30:39.215153 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cdec65986f89463578aabf9ff0a1b1f818b048b74925193184dc1a3f97eef06\": container with ID starting with 4cdec65986f89463578aabf9ff0a1b1f818b048b74925193184dc1a3f97eef06 not found: ID does not exist" containerID="4cdec65986f89463578aabf9ff0a1b1f818b048b74925193184dc1a3f97eef06" Nov 02 18:30:39.215273 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.215187 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cdec65986f89463578aabf9ff0a1b1f818b048b74925193184dc1a3f97eef06"} err="failed to get container status \"4cdec65986f89463578aabf9ff0a1b1f818b048b74925193184dc1a3f97eef06\": rpc error: code = NotFound desc = could not find container \"4cdec65986f89463578aabf9ff0a1b1f818b048b74925193184dc1a3f97eef06\": container with ID starting with 4cdec65986f89463578aabf9ff0a1b1f818b048b74925193184dc1a3f97eef06 not found: ID does not exist" Nov 02 18:30:39.218319 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.218295 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg"] Nov 02 18:30:39.221081 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.221062 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a37cf0-64c6d9bc7-k9mzg"] Nov 02 18:30:39.233695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.233511 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw"] Nov 02 18:30:39.236879 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.236855 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-rdkcw"] Nov 02 18:30:39.426243 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.426207 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28611469-b9c9-435a-87e1-1b7b37f0c5ff" path="/var/lib/kubelet/pods/28611469-b9c9-435a-87e1-1b7b37f0c5ff/volumes" Nov 02 18:30:39.426917 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:39.426896 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee0d3ab7-7076-4998-b997-2fa88e7622ad" path="/var/lib/kubelet/pods/ee0d3ab7-7076-4998-b997-2fa88e7622ad/volumes" Nov 02 18:30:44.147742 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:44.144995 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7"] Nov 02 18:30:44.147742 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:44.145272 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" podUID="42ac5b40-0dae-41ea-a7ca-c7c2313d692f" containerName="istio-proxy" containerID="cri-o://18a062f47e0feba21bc4fd6f27807a70bf20f148b5e9e208bc41c1cb99e72687" gracePeriod=30 Nov 02 18:30:49.272247 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.272185 5601 generic.go:356] "Generic (PLEG): container finished" podID="42ac5b40-0dae-41ea-a7ca-c7c2313d692f" containerID="18a062f47e0feba21bc4fd6f27807a70bf20f148b5e9e208bc41c1cb99e72687" exitCode=0 Nov 02 18:30:49.272611 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.272270 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" event={"ID":"42ac5b40-0dae-41ea-a7ca-c7c2313d692f","Type":"ContainerDied","Data":"18a062f47e0feba21bc4fd6f27807a70bf20f148b5e9e208bc41c1cb99e72687"} Nov 02 18:30:49.420085 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.420058 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 18:30:49.564971 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.564927 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-podinfo\") pod \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " Nov 02 18:30:49.565184 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.564997 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-workload-socket\") pod \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " Nov 02 18:30:49.565184 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.565033 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-envoy\") pod \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " Nov 02 18:30:49.565184 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.565072 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-kube-api-access-gjfvz\" (UniqueName: \"kubernetes.io/projected/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-kube-api-access-gjfvz\") pod \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " Nov 02 18:30:49.565184 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.565105 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istiod-ca-cert\") pod \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " Nov 02 18:30:49.565184 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.565123 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-workload-certs\") pod \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " Nov 02 18:30:49.565184 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.565153 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-data\") pod \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " Nov 02 18:30:49.566154 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.565288 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-token\" (UniqueName: \"kubernetes.io/projected/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-token\") pod \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " Nov 02 18:30:49.566154 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.565329 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-credential-socket\") pod \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\" (UID: \"42ac5b40-0dae-41ea-a7ca-c7c2313d692f\") " Nov 02 18:30:49.566154 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.565402 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-workload-socket" pod "42ac5b40-0dae-41ea-a7ca-c7c2313d692f" (UID: "42ac5b40-0dae-41ea-a7ca-c7c2313d692f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:49.566154 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.565700 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-workload-certs" pod "42ac5b40-0dae-41ea-a7ca-c7c2313d692f" (UID: "42ac5b40-0dae-41ea-a7ca-c7c2313d692f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:49.566154 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.565874 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-data" pod "42ac5b40-0dae-41ea-a7ca-c7c2313d692f" (UID: "42ac5b40-0dae-41ea-a7ca-c7c2313d692f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:49.566154 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.566000 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:49.566154 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.566026 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:49.566154 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.566042 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:49.566697 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.566187 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-credential-socket" pod "42ac5b40-0dae-41ea-a7ca-c7c2313d692f" (UID: "42ac5b40-0dae-41ea-a7ca-c7c2313d692f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:49.566697 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.565524 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istiod-ca-cert" pod "42ac5b40-0dae-41ea-a7ca-c7c2313d692f" (UID: "42ac5b40-0dae-41ea-a7ca-c7c2313d692f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:49.567501 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.567475 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-kube-api-access-gjfvz" pod "42ac5b40-0dae-41ea-a7ca-c7c2313d692f" (UID: "42ac5b40-0dae-41ea-a7ca-c7c2313d692f"). InnerVolumeSpecName "kube-api-access-gjfvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:49.567648 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.567599 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-envoy" pod "42ac5b40-0dae-41ea-a7ca-c7c2313d692f" (UID: "42ac5b40-0dae-41ea-a7ca-c7c2313d692f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:30:49.567719 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.567675 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-podinfo" pod "42ac5b40-0dae-41ea-a7ca-c7c2313d692f" (UID: "42ac5b40-0dae-41ea-a7ca-c7c2313d692f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:30:49.567914 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.567895 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-token" pod "42ac5b40-0dae-41ea-a7ca-c7c2313d692f" (UID: "42ac5b40-0dae-41ea-a7ca-c7c2313d692f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:49.667649 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.667602 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:49.667649 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.667648 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:49.667649 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.667657 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:49.667892 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.667666 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:49.667892 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.667682 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjfvz\" (UniqueName: \"kubernetes.io/projected/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-kube-api-access-gjfvz\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:49.667892 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:49.667692 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ac5b40-0dae-41ea-a7ca-c7c2313d692f-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:30:50.280826 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.280765 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" Nov 02 18:30:50.280826 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.280809 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7" event={"ID":"42ac5b40-0dae-41ea-a7ca-c7c2313d692f","Type":"ContainerDied","Data":"bb248fdaf3167c756da4d3c2e9315a68c0566ecdb26e1bb9666aaaa5ecf5a104"} Nov 02 18:30:50.281530 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.280880 5601 scope.go:122] "RemoveContainer" containerID="18a062f47e0feba21bc4fd6f27807a70bf20f148b5e9e208bc41c1cb99e72687" Nov 02 18:30:50.309269 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.309243 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7"] Nov 02 18:30:50.314779 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.314755 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-pcbf7"] Nov 02 18:30:50.485739 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.485710 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn"] Nov 02 18:30:50.486184 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486169 5601 cpu_manager.go:414] "containerMap: removing container" podUID="01186ed8-9a15-4802-924a-85dd46e7dc8b" containerName="cleanup" Nov 02 18:30:50.486237 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486186 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01186ed8-9a15-4802-924a-85dd46e7dc8b" containerName="cleanup" Nov 02 18:30:50.486237 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486199 5601 cpu_manager.go:414] "containerMap: removing container" podUID="4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99" containerName="cleanup" Nov 02 18:30:50.486237 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486206 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99" containerName="cleanup" Nov 02 18:30:50.486237 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486220 5601 cpu_manager.go:414] "containerMap: removing container" podUID="01186ed8-9a15-4802-924a-85dd46e7dc8b" containerName="cleanup" Nov 02 18:30:50.486237 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486225 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01186ed8-9a15-4802-924a-85dd46e7dc8b" containerName="cleanup" Nov 02 18:30:50.486237 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486233 5601 cpu_manager.go:414] "containerMap: removing container" podUID="deb3a82f-14e6-4cdf-8fa7-12db317f98f8" containerName="cleanup" Nov 02 18:30:50.486237 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486237 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb3a82f-14e6-4cdf-8fa7-12db317f98f8" containerName="cleanup" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486245 5601 cpu_manager.go:414] "containerMap: removing container" podUID="deb3a82f-14e6-4cdf-8fa7-12db317f98f8" containerName="cleanup" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486250 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb3a82f-14e6-4cdf-8fa7-12db317f98f8" containerName="cleanup" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486259 5601 cpu_manager.go:414] "containerMap: removing container" podUID="28611469-b9c9-435a-87e1-1b7b37f0c5ff" containerName="payload-processing" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486263 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28611469-b9c9-435a-87e1-1b7b37f0c5ff" containerName="payload-processing" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486269 5601 cpu_manager.go:414] "containerMap: removing container" podUID="1b0271e7-cb07-45b2-a349-32f49ce6e442" containerName="maas-api" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486274 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b0271e7-cb07-45b2-a349-32f49ce6e442" containerName="maas-api" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486282 5601 cpu_manager.go:414] "containerMap: removing container" podUID="ac238df9-b947-47d0-a52a-38145c23f61a" containerName="cleanup" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486287 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac238df9-b947-47d0-a52a-38145c23f61a" containerName="cleanup" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486292 5601 cpu_manager.go:414] "containerMap: removing container" podUID="ee0d3ab7-7076-4998-b997-2fa88e7622ad" containerName="istio-proxy" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486297 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee0d3ab7-7076-4998-b997-2fa88e7622ad" containerName="istio-proxy" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486304 5601 cpu_manager.go:414] "containerMap: removing container" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486308 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486315 5601 cpu_manager.go:414] "containerMap: removing container" podUID="abc3edc9-ba31-479b-97f7-5eee71b5301b" containerName="maas-api" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486319 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc3edc9-ba31-479b-97f7-5eee71b5301b" containerName="maas-api" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486324 5601 cpu_manager.go:414] "containerMap: removing container" podUID="ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" containerName="istio-proxy" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486329 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" containerName="istio-proxy" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486343 5601 cpu_manager.go:414] "containerMap: removing container" podUID="ac238df9-b947-47d0-a52a-38145c23f61a" containerName="cleanup" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486347 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac238df9-b947-47d0-a52a-38145c23f61a" containerName="cleanup" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486353 5601 cpu_manager.go:414] "containerMap: removing container" podUID="1d5a3bb4-a17b-41cf-b512-f4de4bd0e471" containerName="cleanup" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486358 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d5a3bb4-a17b-41cf-b512-f4de4bd0e471" containerName="cleanup" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486365 5601 cpu_manager.go:414] "containerMap: removing container" podUID="1d5a3bb4-a17b-41cf-b512-f4de4bd0e471" containerName="cleanup" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486370 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d5a3bb4-a17b-41cf-b512-f4de4bd0e471" containerName="cleanup" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486378 5601 cpu_manager.go:414] "containerMap: removing container" podUID="42ac5b40-0dae-41ea-a7ca-c7c2313d692f" containerName="istio-proxy" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486384 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ac5b40-0dae-41ea-a7ca-c7c2313d692f" containerName="istio-proxy" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486390 5601 cpu_manager.go:414] "containerMap: removing container" podUID="4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99" containerName="cleanup" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486395 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99" containerName="cleanup" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486408 5601 cpu_manager.go:414] "containerMap: removing container" podUID="1d5a3bb4-a17b-41cf-b512-f4de4bd0e471" containerName="cleanup" Nov 02 18:30:50.486452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486412 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d5a3bb4-a17b-41cf-b512-f4de4bd0e471" containerName="cleanup" Nov 02 18:30:50.487314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486490 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac238df9-b947-47d0-a52a-38145c23f61a" containerName="cleanup" Nov 02 18:30:50.487314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486497 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d5a3bb4-a17b-41cf-b512-f4de4bd0e471" containerName="cleanup" Nov 02 18:30:50.487314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486503 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d5a3bb4-a17b-41cf-b512-f4de4bd0e471" containerName="cleanup" Nov 02 18:30:50.487314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486509 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb3a82f-14e6-4cdf-8fa7-12db317f98f8" containerName="cleanup" Nov 02 18:30:50.487314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486515 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ac5b40-0dae-41ea-a7ca-c7c2313d692f" containerName="istio-proxy" Nov 02 18:30:50.487314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486520 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99" containerName="cleanup" Nov 02 18:30:50.487314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486527 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99" containerName="cleanup" Nov 02 18:30:50.487314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486532 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="01186ed8-9a15-4802-924a-85dd46e7dc8b" containerName="cleanup" Nov 02 18:30:50.487314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486538 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="01186ed8-9a15-4802-924a-85dd46e7dc8b" containerName="cleanup" Nov 02 18:30:50.487314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486543 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99" containerName="cleanup" Nov 02 18:30:50.487314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486549 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="01186ed8-9a15-4802-924a-85dd46e7dc8b" containerName="cleanup" Nov 02 18:30:50.487314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486555 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb3a82f-14e6-4cdf-8fa7-12db317f98f8" containerName="cleanup" Nov 02 18:30:50.487314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486559 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="28611469-b9c9-435a-87e1-1b7b37f0c5ff" containerName="payload-processing" Nov 02 18:30:50.487314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486564 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b0271e7-cb07-45b2-a349-32f49ce6e442" containerName="maas-api" Nov 02 18:30:50.487314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486570 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac238df9-b947-47d0-a52a-38145c23f61a" containerName="cleanup" Nov 02 18:30:50.487314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486575 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee0d3ab7-7076-4998-b997-2fa88e7622ad" containerName="istio-proxy" Nov 02 18:30:50.487314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486580 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c749700-fae4-498d-9fb1-38652eec3eef" containerName="main" Nov 02 18:30:50.487314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486585 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc3edc9-ba31-479b-97f7-5eee71b5301b" containerName="maas-api" Nov 02 18:30:50.487314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.486589 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca8be970-ee6a-4fbe-94b1-c63f75fa4fda" containerName="istio-proxy" Nov 02 18:30:50.491224 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.491204 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.500345 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.500306 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-47e1c2-openshift-default-dockercfg-q8nvl\"" Nov 02 18:30:50.501491 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.501465 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn"] Nov 02 18:30:50.578677 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.578564 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-envoy\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.578677 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.578606 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-podinfo\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.578914 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.578702 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-data\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.578914 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.578770 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-772v6\" (UniqueName: \"kubernetes.io/projected/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-kube-api-access-772v6\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.578914 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.578862 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-workload-certs\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.578914 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.578903 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-token\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.579095 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.578933 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-credential-socket\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.579095 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.578978 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istiod-ca-cert\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.579095 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.579022 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-workload-socket\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.679557 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.679524 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istiod-ca-cert\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.679753 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.679564 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-workload-socket\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.679753 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.679631 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-envoy\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.679753 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.679656 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-podinfo\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.679753 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.679730 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-data\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.679959 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.679779 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772v6\" (UniqueName: \"kubernetes.io/projected/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-kube-api-access-772v6\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.679959 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.679848 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-workload-certs\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.679959 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.679891 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-token\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.679959 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.679923 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-credential-socket\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.680208 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.680020 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-workload-socket\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.680434 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.680378 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-credential-socket\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.680434 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.680415 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-data\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.680434 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.680427 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-workload-certs\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.680434 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.680379 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istiod-ca-cert\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.682041 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.682014 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-envoy\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.682285 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.682263 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-podinfo\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.687520 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.687500 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-772v6\" (UniqueName: \"kubernetes.io/projected/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-kube-api-access-772v6\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.687707 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.687687 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-token\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:50.807021 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:50.806993 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:51.089212 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:51.089186 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn"] Nov 02 18:30:51.092141 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:30:51.092108 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a8ce0d_ecf7_4f11_a779_c0b60d3dfcc4.slice/crio-e728da02929724a2b9b6370afe779447132df98b929c749f44aec0af7542b7c2 WatchSource:0}: Error finding container e728da02929724a2b9b6370afe779447132df98b929c749f44aec0af7542b7c2: Status 404 returned error can't find the container with id e728da02929724a2b9b6370afe779447132df98b929c749f44aec0af7542b7c2 Nov 02 18:30:51.094945 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:51.094905 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:30:51.095053 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:51.094979 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:30:51.095053 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:51.095015 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:30:51.290542 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:51.290507 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" event={"ID":"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4","Type":"ContainerStarted","Data":"d59a2c5453a2561c58293a1dd802fcddffddf3c07bfe2bfbfe78b17a7644e462"} Nov 02 18:30:51.290963 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:51.290569 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" event={"ID":"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4","Type":"ContainerStarted","Data":"e728da02929724a2b9b6370afe779447132df98b929c749f44aec0af7542b7c2"} Nov 02 18:30:51.313067 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:51.313005 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" podStartSLOduration=1.31298721 podStartE2EDuration="1.31298721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:51.311020603 +0000 UTC m=+3416.002074615" watchObservedRunningTime="2026-11-02 18:30:51.31298721 +0000 UTC m=+3416.004041222" Nov 02 18:30:51.426278 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:51.426245 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42ac5b40-0dae-41ea-a7ca-c7c2313d692f" path="/var/lib/kubelet/pods/42ac5b40-0dae-41ea-a7ca-c7c2313d692f/volumes" Nov 02 18:30:51.816263 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:51.810053 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:52.809266 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:52.809225 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Nov 02 18:30:52.809794 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:52.809288 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" podUID="48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Nov 02 18:30:53.807360 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:53.807323 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Nov 02 18:30:53.807578 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:53.807385 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" podUID="48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Nov 02 18:30:54.807849 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:54.807808 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Nov 02 18:30:54.808274 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:54.807866 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" podUID="48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Nov 02 18:30:55.808357 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:55.808317 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Nov 02 18:30:55.808795 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:55.808384 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" podUID="48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Nov 02 18:30:56.808317 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:56.808281 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Nov 02 18:30:56.808534 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:56.808344 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" podUID="48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Nov 02 18:30:57.812119 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:57.812088 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:57.812552 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:57.812363 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:30:57.813152 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:30:57.813135 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:31:04.484393 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:31:04.484346 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:31:18.145873 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:18.145834 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl"] Nov 02 18:31:18.146769 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:18.146699 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" podUID="f26c04ee-0060-4863-b50f-2eed99aad8b1" containerName="istio-proxy" containerID="cri-o://f45cbd85089b41a367d6877f7ba0ecb7890b6e565ff12d24e2e1b1aa700e593f" gracePeriod=30 Nov 02 18:31:23.426750 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.426676 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 18:31:23.455048 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.455006 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-workload-certs\") pod \"f26c04ee-0060-4863-b50f-2eed99aad8b1\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " Nov 02 18:31:23.455231 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.455106 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26c04ee-0060-4863-b50f-2eed99aad8b1-kube-api-access-5r6sn\" (UniqueName: \"kubernetes.io/projected/f26c04ee-0060-4863-b50f-2eed99aad8b1-kube-api-access-5r6sn\") pod \"f26c04ee-0060-4863-b50f-2eed99aad8b1\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " Nov 02 18:31:23.455231 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.455137 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-workload-socket\") pod \"f26c04ee-0060-4863-b50f-2eed99aad8b1\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " Nov 02 18:31:23.455231 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.455188 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-podinfo\") pod \"f26c04ee-0060-4863-b50f-2eed99aad8b1\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " Nov 02 18:31:23.455231 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.455226 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-token\" (UniqueName: \"kubernetes.io/projected/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-token\") pod \"f26c04ee-0060-4863-b50f-2eed99aad8b1\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " Nov 02 18:31:23.455385 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.455306 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-envoy\") pod \"f26c04ee-0060-4863-b50f-2eed99aad8b1\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " Nov 02 18:31:23.455385 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.455338 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-workload-certs" pod "f26c04ee-0060-4863-b50f-2eed99aad8b1" (UID: "f26c04ee-0060-4863-b50f-2eed99aad8b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:31:23.455385 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.455355 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-data\") pod \"f26c04ee-0060-4863-b50f-2eed99aad8b1\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " Nov 02 18:31:23.455505 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.455383 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-credential-socket\") pod \"f26c04ee-0060-4863-b50f-2eed99aad8b1\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " Nov 02 18:31:23.455505 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.455487 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f26c04ee-0060-4863-b50f-2eed99aad8b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26c04ee-0060-4863-b50f-2eed99aad8b1-istiod-ca-cert\") pod \"f26c04ee-0060-4863-b50f-2eed99aad8b1\" (UID: \"f26c04ee-0060-4863-b50f-2eed99aad8b1\") " Nov 02 18:31:23.456195 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.455992 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:31:23.456195 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.456003 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-credential-socket" pod "f26c04ee-0060-4863-b50f-2eed99aad8b1" (UID: "f26c04ee-0060-4863-b50f-2eed99aad8b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:31:23.456195 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.456086 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f26c04ee-0060-4863-b50f-2eed99aad8b1-istiod-ca-cert" pod "f26c04ee-0060-4863-b50f-2eed99aad8b1" (UID: "f26c04ee-0060-4863-b50f-2eed99aad8b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:31:23.456568 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.456505 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-data" pod "f26c04ee-0060-4863-b50f-2eed99aad8b1" (UID: "f26c04ee-0060-4863-b50f-2eed99aad8b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:31:23.457783 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.457755 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-podinfo" pod "f26c04ee-0060-4863-b50f-2eed99aad8b1" (UID: "f26c04ee-0060-4863-b50f-2eed99aad8b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:31:23.458136 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.458108 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-envoy" pod "f26c04ee-0060-4863-b50f-2eed99aad8b1" (UID: "f26c04ee-0060-4863-b50f-2eed99aad8b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:31:23.458255 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.458192 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26c04ee-0060-4863-b50f-2eed99aad8b1-kube-api-access-5r6sn" pod "f26c04ee-0060-4863-b50f-2eed99aad8b1" (UID: "f26c04ee-0060-4863-b50f-2eed99aad8b1"). InnerVolumeSpecName "kube-api-access-5r6sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:31:23.459121 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.459097 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-workload-socket" pod "f26c04ee-0060-4863-b50f-2eed99aad8b1" (UID: "f26c04ee-0060-4863-b50f-2eed99aad8b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:31:23.461025 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.461003 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-token" pod "f26c04ee-0060-4863-b50f-2eed99aad8b1" (UID: "f26c04ee-0060-4863-b50f-2eed99aad8b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:31:23.493917 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.493524 5601 generic.go:356] "Generic (PLEG): container finished" podID="f26c04ee-0060-4863-b50f-2eed99aad8b1" containerID="f45cbd85089b41a367d6877f7ba0ecb7890b6e565ff12d24e2e1b1aa700e593f" exitCode=0 Nov 02 18:31:23.493917 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.493592 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" Nov 02 18:31:23.493917 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.493724 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" event={"ID":"f26c04ee-0060-4863-b50f-2eed99aad8b1","Type":"ContainerDied","Data":"f45cbd85089b41a367d6877f7ba0ecb7890b6e565ff12d24e2e1b1aa700e593f"} Nov 02 18:31:23.493917 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.493765 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl" event={"ID":"f26c04ee-0060-4863-b50f-2eed99aad8b1","Type":"ContainerDied","Data":"90322c0dabf0d9cc4407a6e750fb863d9b20ab3fbc5719975f1ca4dd6e1fe938"} Nov 02 18:31:23.493917 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.493790 5601 scope.go:122] "RemoveContainer" containerID="f45cbd85089b41a367d6877f7ba0ecb7890b6e565ff12d24e2e1b1aa700e593f" Nov 02 18:31:23.505984 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.505954 5601 scope.go:122] "RemoveContainer" containerID="857d53115e96c7bc173b8372ff4b8b85ab10357ee05cb82a85f0c29bc5b55d53" Nov 02 18:31:23.515799 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.515771 5601 scope.go:122] "RemoveContainer" containerID="f45cbd85089b41a367d6877f7ba0ecb7890b6e565ff12d24e2e1b1aa700e593f" Nov 02 18:31:23.516112 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:31:23.516094 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f45cbd85089b41a367d6877f7ba0ecb7890b6e565ff12d24e2e1b1aa700e593f\": container with ID starting with f45cbd85089b41a367d6877f7ba0ecb7890b6e565ff12d24e2e1b1aa700e593f not found: ID does not exist" containerID="f45cbd85089b41a367d6877f7ba0ecb7890b6e565ff12d24e2e1b1aa700e593f" Nov 02 18:31:23.516180 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.516124 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f45cbd85089b41a367d6877f7ba0ecb7890b6e565ff12d24e2e1b1aa700e593f"} err="failed to get container status \"f45cbd85089b41a367d6877f7ba0ecb7890b6e565ff12d24e2e1b1aa700e593f\": rpc error: code = NotFound desc = could not find container \"f45cbd85089b41a367d6877f7ba0ecb7890b6e565ff12d24e2e1b1aa700e593f\": container with ID starting with f45cbd85089b41a367d6877f7ba0ecb7890b6e565ff12d24e2e1b1aa700e593f not found: ID does not exist" Nov 02 18:31:23.516180 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.516142 5601 scope.go:122] "RemoveContainer" containerID="857d53115e96c7bc173b8372ff4b8b85ab10357ee05cb82a85f0c29bc5b55d53" Nov 02 18:31:23.516449 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:31:23.516424 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857d53115e96c7bc173b8372ff4b8b85ab10357ee05cb82a85f0c29bc5b55d53\": container with ID starting with 857d53115e96c7bc173b8372ff4b8b85ab10357ee05cb82a85f0c29bc5b55d53 not found: ID does not exist" containerID="857d53115e96c7bc173b8372ff4b8b85ab10357ee05cb82a85f0c29bc5b55d53" Nov 02 18:31:23.516511 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.516458 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857d53115e96c7bc173b8372ff4b8b85ab10357ee05cb82a85f0c29bc5b55d53"} err="failed to get container status \"857d53115e96c7bc173b8372ff4b8b85ab10357ee05cb82a85f0c29bc5b55d53\": rpc error: code = NotFound desc = could not find container \"857d53115e96c7bc173b8372ff4b8b85ab10357ee05cb82a85f0c29bc5b55d53\": container with ID starting with 857d53115e96c7bc173b8372ff4b8b85ab10357ee05cb82a85f0c29bc5b55d53 not found: ID does not exist" Nov 02 18:31:23.521491 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.520920 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl"] Nov 02 18:31:23.524175 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.524147 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-qnsxl"] Nov 02 18:31:23.561863 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.561671 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r6sn\" (UniqueName: \"kubernetes.io/projected/f26c04ee-0060-4863-b50f-2eed99aad8b1-kube-api-access-5r6sn\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:31:23.561863 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.561719 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:31:23.561863 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.561737 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:31:23.561863 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.561752 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:31:23.561863 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.561769 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:31:23.561863 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.561784 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:31:23.561863 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.561800 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c04ee-0060-4863-b50f-2eed99aad8b1-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:31:23.561863 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:23.561817 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26c04ee-0060-4863-b50f-2eed99aad8b1-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:31:25.426850 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:31:25.426808 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f26c04ee-0060-4863-b50f-2eed99aad8b1" path="/var/lib/kubelet/pods/f26c04ee-0060-4863-b50f-2eed99aad8b1/volumes" Nov 02 18:31:34.484038 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:31:34.484002 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:32:04.485823 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:32:04.485778 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:32:34.485144 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:32:34.485105 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:32:37.126871 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:37.126838 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r"] Nov 02 18:32:52.150640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.149876 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" podUID="d098a18f-93d8-4958-833b-252443a3b8b8" containerName="main" containerID="cri-o://47d38c97c644a46e25664e679daf090967ac4502249994f273c83de5e81b929f" gracePeriod=45 Nov 02 18:32:52.442695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.442668 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:32:52.482075 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.482040 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-home\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-home\") pod \"d098a18f-93d8-4958-833b-252443a3b8b8\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " Nov 02 18:32:52.482075 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.482075 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-dshm\") pod \"d098a18f-93d8-4958-833b-252443a3b8b8\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " Nov 02 18:32:52.482329 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.482102 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-tmp-dir\") pod \"d098a18f-93d8-4958-833b-252443a3b8b8\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " Nov 02 18:32:52.482329 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.482147 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-model-cache\") pod \"d098a18f-93d8-4958-833b-252443a3b8b8\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " Nov 02 18:32:52.482329 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.482171 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d098a18f-93d8-4958-833b-252443a3b8b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/d098a18f-93d8-4958-833b-252443a3b8b8-tls-certs\") pod \"d098a18f-93d8-4958-833b-252443a3b8b8\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " Nov 02 18:32:52.482329 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.482283 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d098a18f-93d8-4958-833b-252443a3b8b8-kube-api-access-kjl78\" (UniqueName: \"kubernetes.io/projected/d098a18f-93d8-4958-833b-252443a3b8b8-kube-api-access-kjl78\") pod \"d098a18f-93d8-4958-833b-252443a3b8b8\" (UID: \"d098a18f-93d8-4958-833b-252443a3b8b8\") " Nov 02 18:32:52.482560 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.482367 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-home" pod "d098a18f-93d8-4958-833b-252443a3b8b8" (UID: "d098a18f-93d8-4958-833b-252443a3b8b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:32:52.482560 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.482443 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-model-cache" pod "d098a18f-93d8-4958-833b-252443a3b8b8" (UID: "d098a18f-93d8-4958-833b-252443a3b8b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:32:52.482835 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.482814 5601 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-home\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:52.482925 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.482840 5601 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-model-cache\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:52.484377 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.484339 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-tmp-dir" pod "d098a18f-93d8-4958-833b-252443a3b8b8" (UID: "d098a18f-93d8-4958-833b-252443a3b8b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:32:52.486321 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.486287 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d098a18f-93d8-4958-833b-252443a3b8b8-kube-api-access-kjl78" pod "d098a18f-93d8-4958-833b-252443a3b8b8" (UID: "d098a18f-93d8-4958-833b-252443a3b8b8"). InnerVolumeSpecName "kube-api-access-kjl78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:32:52.486456 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.486435 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d098a18f-93d8-4958-833b-252443a3b8b8-tls-certs" pod "d098a18f-93d8-4958-833b-252443a3b8b8" (UID: "d098a18f-93d8-4958-833b-252443a3b8b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:32:52.494771 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.494739 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-dshm" pod "d098a18f-93d8-4958-833b-252443a3b8b8" (UID: "d098a18f-93d8-4958-833b-252443a3b8b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:32:52.583900 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.583862 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjl78\" (UniqueName: \"kubernetes.io/projected/d098a18f-93d8-4958-833b-252443a3b8b8-kube-api-access-kjl78\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:52.583900 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.583898 5601 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-dshm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:52.584106 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.583913 5601 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d098a18f-93d8-4958-833b-252443a3b8b8-tmp-dir\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:52.584106 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:52.583925 5601 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d098a18f-93d8-4958-833b-252443a3b8b8-tls-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:32:53.095766 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:53.095724 5601 generic.go:356] "Generic (PLEG): container finished" podID="d098a18f-93d8-4958-833b-252443a3b8b8" containerID="47d38c97c644a46e25664e679daf090967ac4502249994f273c83de5e81b929f" exitCode=0 Nov 02 18:32:53.095936 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:53.095823 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" Nov 02 18:32:53.095936 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:53.095859 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" event={"ID":"d098a18f-93d8-4958-833b-252443a3b8b8","Type":"ContainerDied","Data":"47d38c97c644a46e25664e679daf090967ac4502249994f273c83de5e81b929f"} Nov 02 18:32:53.095936 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:53.095902 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r" event={"ID":"d098a18f-93d8-4958-833b-252443a3b8b8","Type":"ContainerDied","Data":"e9014e926473b0c77c2a8233638476f7486cebbdbdc564534e9bc2a0d71f8643"} Nov 02 18:32:53.095936 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:53.095919 5601 scope.go:122] "RemoveContainer" containerID="47d38c97c644a46e25664e679daf090967ac4502249994f273c83de5e81b929f" Nov 02 18:32:53.106937 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:53.106916 5601 scope.go:122] "RemoveContainer" containerID="47d38c97c644a46e25664e679daf090967ac4502249994f273c83de5e81b929f" Nov 02 18:32:53.107243 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:32:53.107216 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d38c97c644a46e25664e679daf090967ac4502249994f273c83de5e81b929f\": container with ID starting with 47d38c97c644a46e25664e679daf090967ac4502249994f273c83de5e81b929f not found: ID does not exist" containerID="47d38c97c644a46e25664e679daf090967ac4502249994f273c83de5e81b929f" Nov 02 18:32:53.107347 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:53.107250 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d38c97c644a46e25664e679daf090967ac4502249994f273c83de5e81b929f"} err="failed to get container status \"47d38c97c644a46e25664e679daf090967ac4502249994f273c83de5e81b929f\": rpc error: code = NotFound desc = could not find container \"47d38c97c644a46e25664e679daf090967ac4502249994f273c83de5e81b929f\": container with ID starting with 47d38c97c644a46e25664e679daf090967ac4502249994f273c83de5e81b929f not found: ID does not exist" Nov 02 18:32:53.119550 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:53.119527 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r"] Nov 02 18:32:53.126709 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:53.126682 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7771bf/sub-test-model-7771bf-kserve-7cc9b46d9c-fg55r"] Nov 02 18:32:53.427339 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:32:53.427299 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d098a18f-93d8-4958-833b-252443a3b8b8" path="/var/lib/kubelet/pods/d098a18f-93d8-4958-833b-252443a3b8b8/volumes" Nov 02 18:33:04.484115 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:33:04.484078 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:33:08.145826 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:08.145785 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh"] Nov 02 18:33:08.146481 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:08.146053 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" podUID="ad96fe9c-98b9-41bb-b532-1b3ca4da572c" containerName="istio-proxy" containerID="cri-o://0ae24df59b04dd5729bb916723d67cbc9466e2fc783fe2a3df269adbbb2e2205" gracePeriod=30 Nov 02 18:33:13.409002 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.408974 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:33:13.517311 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.517275 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istiod-ca-cert\") pod \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " Nov 02 18:33:13.517504 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.517367 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-credential-socket\") pod \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " Nov 02 18:33:13.517504 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.517396 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-podinfo\") pod \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " Nov 02 18:33:13.517504 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.517440 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-token\" (UniqueName: \"kubernetes.io/projected/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-token\") pod \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " Nov 02 18:33:13.517504 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.517491 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-kube-api-access-ns9cx\" (UniqueName: \"kubernetes.io/projected/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-kube-api-access-ns9cx\") pod \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " Nov 02 18:33:13.517796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.517532 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istiod-ca-cert" pod "ad96fe9c-98b9-41bb-b532-1b3ca4da572c" (UID: "ad96fe9c-98b9-41bb-b532-1b3ca4da572c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:33:13.517796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.517554 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-workload-socket\") pod \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " Nov 02 18:33:13.517796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.517609 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-envoy\") pod \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " Nov 02 18:33:13.517796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.517678 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-workload-certs\") pod \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " Nov 02 18:33:13.517796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.517716 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-credential-socket" pod "ad96fe9c-98b9-41bb-b532-1b3ca4da572c" (UID: "ad96fe9c-98b9-41bb-b532-1b3ca4da572c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:13.517796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.517735 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-data\") pod \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\" (UID: \"ad96fe9c-98b9-41bb-b532-1b3ca4da572c\") " Nov 02 18:33:13.518132 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.517896 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-workload-socket" pod "ad96fe9c-98b9-41bb-b532-1b3ca4da572c" (UID: "ad96fe9c-98b9-41bb-b532-1b3ca4da572c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:13.518132 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.517927 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-workload-certs" pod "ad96fe9c-98b9-41bb-b532-1b3ca4da572c" (UID: "ad96fe9c-98b9-41bb-b532-1b3ca4da572c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:13.518250 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.518139 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-data" pod "ad96fe9c-98b9-41bb-b532-1b3ca4da572c" (UID: "ad96fe9c-98b9-41bb-b532-1b3ca4da572c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:13.519508 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.519486 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:13.519715 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.519699 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:13.519839 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.519826 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:13.519963 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.519940 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:13.520091 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.519965 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:13.520232 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.520199 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-podinfo" pod "ad96fe9c-98b9-41bb-b532-1b3ca4da572c" (UID: "ad96fe9c-98b9-41bb-b532-1b3ca4da572c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:33:13.520325 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.520232 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-kube-api-access-ns9cx" pod "ad96fe9c-98b9-41bb-b532-1b3ca4da572c" (UID: "ad96fe9c-98b9-41bb-b532-1b3ca4da572c"). InnerVolumeSpecName "kube-api-access-ns9cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:13.520494 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.520473 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-token" pod "ad96fe9c-98b9-41bb-b532-1b3ca4da572c" (UID: "ad96fe9c-98b9-41bb-b532-1b3ca4da572c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:13.520602 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.520584 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-envoy" pod "ad96fe9c-98b9-41bb-b532-1b3ca4da572c" (UID: "ad96fe9c-98b9-41bb-b532-1b3ca4da572c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:13.620896 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.620859 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:13.620896 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.620890 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:13.620896 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.620900 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:13.621223 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:13.620909 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns9cx\" (UniqueName: \"kubernetes.io/projected/ad96fe9c-98b9-41bb-b532-1b3ca4da572c-kube-api-access-ns9cx\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:14.228788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:14.228750 5601 generic.go:356] "Generic (PLEG): container finished" podID="ad96fe9c-98b9-41bb-b532-1b3ca4da572c" containerID="0ae24df59b04dd5729bb916723d67cbc9466e2fc783fe2a3df269adbbb2e2205" exitCode=0 Nov 02 18:33:14.228985 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:14.228811 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" Nov 02 18:33:14.228985 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:14.228840 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" event={"ID":"ad96fe9c-98b9-41bb-b532-1b3ca4da572c","Type":"ContainerDied","Data":"0ae24df59b04dd5729bb916723d67cbc9466e2fc783fe2a3df269adbbb2e2205"} Nov 02 18:33:14.228985 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:14.228904 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh" event={"ID":"ad96fe9c-98b9-41bb-b532-1b3ca4da572c","Type":"ContainerDied","Data":"9b3b6bd9f2c2c1aef557892795c21ac3670130fe1342b1add9f2610c3a9b4703"} Nov 02 18:33:14.228985 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:14.228935 5601 scope.go:122] "RemoveContainer" containerID="0ae24df59b04dd5729bb916723d67cbc9466e2fc783fe2a3df269adbbb2e2205" Nov 02 18:33:14.239444 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:14.239419 5601 scope.go:122] "RemoveContainer" containerID="9682a965c39fdc4f1051f46b8adf3adfe9b24ebca419ea815e91564751689a46" Nov 02 18:33:14.248657 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:14.248634 5601 scope.go:122] "RemoveContainer" containerID="0ae24df59b04dd5729bb916723d67cbc9466e2fc783fe2a3df269adbbb2e2205" Nov 02 18:33:14.248979 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:33:14.248952 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae24df59b04dd5729bb916723d67cbc9466e2fc783fe2a3df269adbbb2e2205\": container with ID starting with 0ae24df59b04dd5729bb916723d67cbc9466e2fc783fe2a3df269adbbb2e2205 not found: ID does not exist" containerID="0ae24df59b04dd5729bb916723d67cbc9466e2fc783fe2a3df269adbbb2e2205" Nov 02 18:33:14.249085 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:14.248996 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae24df59b04dd5729bb916723d67cbc9466e2fc783fe2a3df269adbbb2e2205"} err="failed to get container status \"0ae24df59b04dd5729bb916723d67cbc9466e2fc783fe2a3df269adbbb2e2205\": rpc error: code = NotFound desc = could not find container \"0ae24df59b04dd5729bb916723d67cbc9466e2fc783fe2a3df269adbbb2e2205\": container with ID starting with 0ae24df59b04dd5729bb916723d67cbc9466e2fc783fe2a3df269adbbb2e2205 not found: ID does not exist" Nov 02 18:33:14.249085 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:14.249019 5601 scope.go:122] "RemoveContainer" containerID="9682a965c39fdc4f1051f46b8adf3adfe9b24ebca419ea815e91564751689a46" Nov 02 18:33:14.250417 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:33:14.249932 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9682a965c39fdc4f1051f46b8adf3adfe9b24ebca419ea815e91564751689a46\": container with ID starting with 9682a965c39fdc4f1051f46b8adf3adfe9b24ebca419ea815e91564751689a46 not found: ID does not exist" containerID="9682a965c39fdc4f1051f46b8adf3adfe9b24ebca419ea815e91564751689a46" Nov 02 18:33:14.250417 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:14.249962 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9682a965c39fdc4f1051f46b8adf3adfe9b24ebca419ea815e91564751689a46"} err="failed to get container status \"9682a965c39fdc4f1051f46b8adf3adfe9b24ebca419ea815e91564751689a46\": rpc error: code = NotFound desc = could not find container \"9682a965c39fdc4f1051f46b8adf3adfe9b24ebca419ea815e91564751689a46\": container with ID starting with 9682a965c39fdc4f1051f46b8adf3adfe9b24ebca419ea815e91564751689a46 not found: ID does not exist" Nov 02 18:33:14.252395 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:14.252376 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh"] Nov 02 18:33:14.255788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:14.255758 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-2db162-openshift-default-56b7cbc657-pw6qh"] Nov 02 18:33:15.425600 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:15.425566 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad96fe9c-98b9-41bb-b532-1b3ca4da572c" path="/var/lib/kubelet/pods/ad96fe9c-98b9-41bb-b532-1b3ca4da572c/volumes" Nov 02 18:33:22.398310 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:22.398272 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2"] Nov 02 18:33:22.398731 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:22.398488 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" podUID="e30e6af9-02a5-4b4f-900a-b1de3b3daae9" containerName="istio-proxy" containerID="cri-o://194b37023780e9409594a1e4c62849f7139dffa768b77538d52e75ffeb421a1c" gracePeriod=30 Nov 02 18:33:25.727916 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:25.727866 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:33:25.728386 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:25.727938 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" podUID="e30e6af9-02a5-4b4f-900a-b1de3b3daae9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:33:27.659810 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.659783 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 18:33:27.703304 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.703267 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istiod-ca-cert\") pod \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " Nov 02 18:33:27.703493 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.703380 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-workload-certs\") pod \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " Nov 02 18:33:27.703493 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.703420 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-workload-socket\") pod \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " Nov 02 18:33:27.703493 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.703466 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-kube-api-access-qb8zw\" (UniqueName: \"kubernetes.io/projected/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-kube-api-access-qb8zw\") pod \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " Nov 02 18:33:27.703718 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.703519 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-envoy\") pod \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " Nov 02 18:33:27.703718 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.703566 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-credential-socket\") pod \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " Nov 02 18:33:27.703718 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.703595 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-data\") pod \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " Nov 02 18:33:27.703718 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.703671 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-podinfo\") pod \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " Nov 02 18:33:27.703951 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.703753 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-token\" (UniqueName: \"kubernetes.io/projected/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-token\") pod \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\" (UID: \"e30e6af9-02a5-4b4f-900a-b1de3b3daae9\") " Nov 02 18:33:27.703951 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.703768 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-workload-certs" pod "e30e6af9-02a5-4b4f-900a-b1de3b3daae9" (UID: "e30e6af9-02a5-4b4f-900a-b1de3b3daae9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:27.703951 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.703810 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istiod-ca-cert" pod "e30e6af9-02a5-4b4f-900a-b1de3b3daae9" (UID: "e30e6af9-02a5-4b4f-900a-b1de3b3daae9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:33:27.703951 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.703929 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-credential-socket" pod "e30e6af9-02a5-4b4f-900a-b1de3b3daae9" (UID: "e30e6af9-02a5-4b4f-900a-b1de3b3daae9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:27.704173 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.704096 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-data" pod "e30e6af9-02a5-4b4f-900a-b1de3b3daae9" (UID: "e30e6af9-02a5-4b4f-900a-b1de3b3daae9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:27.704340 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.704319 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:27.704439 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.704344 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:27.704439 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.704359 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:27.704439 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.704374 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:27.704439 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.704434 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-workload-socket" pod "e30e6af9-02a5-4b4f-900a-b1de3b3daae9" (UID: "e30e6af9-02a5-4b4f-900a-b1de3b3daae9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:27.706065 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.706041 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-kube-api-access-qb8zw" pod "e30e6af9-02a5-4b4f-900a-b1de3b3daae9" (UID: "e30e6af9-02a5-4b4f-900a-b1de3b3daae9"). InnerVolumeSpecName "kube-api-access-qb8zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:27.706254 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.706234 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-envoy" pod "e30e6af9-02a5-4b4f-900a-b1de3b3daae9" (UID: "e30e6af9-02a5-4b4f-900a-b1de3b3daae9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:27.706490 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.706469 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-podinfo" pod "e30e6af9-02a5-4b4f-900a-b1de3b3daae9" (UID: "e30e6af9-02a5-4b4f-900a-b1de3b3daae9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:33:27.706569 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.706547 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-token" pod "e30e6af9-02a5-4b4f-900a-b1de3b3daae9" (UID: "e30e6af9-02a5-4b4f-900a-b1de3b3daae9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:27.805312 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.805266 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:27.805312 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.805298 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:27.805312 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.805307 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:27.805312 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.805317 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb8zw\" (UniqueName: \"kubernetes.io/projected/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-kube-api-access-qb8zw\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:27.805312 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:27.805327 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e30e6af9-02a5-4b4f-900a-b1de3b3daae9-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:28.317449 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:28.317409 5601 generic.go:356] "Generic (PLEG): container finished" podID="e30e6af9-02a5-4b4f-900a-b1de3b3daae9" containerID="194b37023780e9409594a1e4c62849f7139dffa768b77538d52e75ffeb421a1c" exitCode=0 Nov 02 18:33:28.317647 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:28.317461 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" Nov 02 18:33:28.317647 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:28.317493 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" event={"ID":"e30e6af9-02a5-4b4f-900a-b1de3b3daae9","Type":"ContainerDied","Data":"194b37023780e9409594a1e4c62849f7139dffa768b77538d52e75ffeb421a1c"} Nov 02 18:33:28.317647 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:28.317551 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2" event={"ID":"e30e6af9-02a5-4b4f-900a-b1de3b3daae9","Type":"ContainerDied","Data":"f46fa36f3897210d27a936c66bf1423cf701f49c27a1146d5a10b45419ab9435"} Nov 02 18:33:28.317647 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:28.317573 5601 scope.go:122] "RemoveContainer" containerID="194b37023780e9409594a1e4c62849f7139dffa768b77538d52e75ffeb421a1c" Nov 02 18:33:28.328723 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:28.328598 5601 scope.go:122] "RemoveContainer" containerID="194b37023780e9409594a1e4c62849f7139dffa768b77538d52e75ffeb421a1c" Nov 02 18:33:28.328986 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:33:28.328965 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"194b37023780e9409594a1e4c62849f7139dffa768b77538d52e75ffeb421a1c\": container with ID starting with 194b37023780e9409594a1e4c62849f7139dffa768b77538d52e75ffeb421a1c not found: ID does not exist" containerID="194b37023780e9409594a1e4c62849f7139dffa768b77538d52e75ffeb421a1c" Nov 02 18:33:28.329040 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:28.328994 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"194b37023780e9409594a1e4c62849f7139dffa768b77538d52e75ffeb421a1c"} err="failed to get container status \"194b37023780e9409594a1e4c62849f7139dffa768b77538d52e75ffeb421a1c\": rpc error: code = NotFound desc = could not find container \"194b37023780e9409594a1e4c62849f7139dffa768b77538d52e75ffeb421a1c\": container with ID starting with 194b37023780e9409594a1e4c62849f7139dffa768b77538d52e75ffeb421a1c not found: ID does not exist" Nov 02 18:33:28.349651 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:28.349592 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2"] Nov 02 18:33:28.349858 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:28.349675 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-rt2c2"] Nov 02 18:33:29.428206 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:29.428156 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e30e6af9-02a5-4b4f-900a-b1de3b3daae9" path="/var/lib/kubelet/pods/e30e6af9-02a5-4b4f-900a-b1de3b3daae9/volumes" Nov 02 18:33:34.484708 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:33:34.484673 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:33:43.426020 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:43.425990 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl"] Nov 02 18:33:43.426707 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:43.426038 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz"] Nov 02 18:33:43.448953 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:43.448922 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5"] Nov 02 18:33:51.244505 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:51.243665 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8"] Nov 02 18:33:51.245342 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:51.245291 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" podUID="13f70b10-e5fb-4844-b55d-5883abbfd8e5" containerName="maas-api" containerID="cri-o://5674dd33ce9268848c0406005e36739a44d0dd97ef676b20384cbfb5fb25bf9a" gracePeriod=30 Nov 02 18:33:51.511688 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:51.511382 5601 generic.go:356] "Generic (PLEG): container finished" podID="13f70b10-e5fb-4844-b55d-5883abbfd8e5" containerID="5674dd33ce9268848c0406005e36739a44d0dd97ef676b20384cbfb5fb25bf9a" exitCode=0 Nov 02 18:33:51.511688 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:51.511470 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" event={"ID":"13f70b10-e5fb-4844-b55d-5883abbfd8e5","Type":"ContainerDied","Data":"5674dd33ce9268848c0406005e36739a44d0dd97ef676b20384cbfb5fb25bf9a"} Nov 02 18:33:51.565259 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:51.565229 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" Nov 02 18:33:51.607796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:51.607756 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13f70b10-e5fb-4844-b55d-5883abbfd8e5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f70b10-e5fb-4844-b55d-5883abbfd8e5-openshift-service-ca\") pod \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\" (UID: \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\") " Nov 02 18:33:51.607993 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:51.607842 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13f70b10-e5fb-4844-b55d-5883abbfd8e5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13f70b10-e5fb-4844-b55d-5883abbfd8e5-metrics-certs\") pod \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\" (UID: \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\") " Nov 02 18:33:51.607993 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:51.607901 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f70b10-e5fb-4844-b55d-5883abbfd8e5-kube-api-access-8c8vp\" (UniqueName: \"kubernetes.io/projected/13f70b10-e5fb-4844-b55d-5883abbfd8e5-kube-api-access-8c8vp\") pod \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\" (UID: \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\") " Nov 02 18:33:51.608128 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:51.608014 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13f70b10-e5fb-4844-b55d-5883abbfd8e5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13f70b10-e5fb-4844-b55d-5883abbfd8e5-maas-api-tls\") pod \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\" (UID: \"13f70b10-e5fb-4844-b55d-5883abbfd8e5\") " Nov 02 18:33:51.608241 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:51.608200 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f70b10-e5fb-4844-b55d-5883abbfd8e5-openshift-service-ca" pod "13f70b10-e5fb-4844-b55d-5883abbfd8e5" (UID: "13f70b10-e5fb-4844-b55d-5883abbfd8e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:33:51.608535 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:51.608511 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f70b10-e5fb-4844-b55d-5883abbfd8e5-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:51.610553 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:51.610522 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f70b10-e5fb-4844-b55d-5883abbfd8e5-kube-api-access-8c8vp" pod "13f70b10-e5fb-4844-b55d-5883abbfd8e5" (UID: "13f70b10-e5fb-4844-b55d-5883abbfd8e5"). InnerVolumeSpecName "kube-api-access-8c8vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:51.610683 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:51.610550 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f70b10-e5fb-4844-b55d-5883abbfd8e5-metrics-certs" pod "13f70b10-e5fb-4844-b55d-5883abbfd8e5" (UID: "13f70b10-e5fb-4844-b55d-5883abbfd8e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:33:51.610767 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:51.610746 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f70b10-e5fb-4844-b55d-5883abbfd8e5-maas-api-tls" pod "13f70b10-e5fb-4844-b55d-5883abbfd8e5" (UID: "13f70b10-e5fb-4844-b55d-5883abbfd8e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:33:51.709871 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:51.709823 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13f70b10-e5fb-4844-b55d-5883abbfd8e5-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:51.709871 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:51.709866 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13f70b10-e5fb-4844-b55d-5883abbfd8e5-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:51.709871 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:51.709878 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c8vp\" (UniqueName: \"kubernetes.io/projected/13f70b10-e5fb-4844-b55d-5883abbfd8e5-kube-api-access-8c8vp\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:52.518057 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:52.518024 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" Nov 02 18:33:52.518510 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:52.518059 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8" event={"ID":"13f70b10-e5fb-4844-b55d-5883abbfd8e5","Type":"ContainerDied","Data":"079a7d0ff3039bf90e22c48dd8acab636f9385d7242e34aa855bccf20c4b186d"} Nov 02 18:33:52.518510 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:52.518123 5601 scope.go:122] "RemoveContainer" containerID="5674dd33ce9268848c0406005e36739a44d0dd97ef676b20384cbfb5fb25bf9a" Nov 02 18:33:52.542875 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:52.542847 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8"] Nov 02 18:33:52.546498 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:52.546472 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5ec436-7f974bbb94-hrbk8"] Nov 02 18:33:53.432483 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:53.432445 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13f70b10-e5fb-4844-b55d-5883abbfd8e5" path="/var/lib/kubelet/pods/13f70b10-e5fb-4844-b55d-5883abbfd8e5/volumes" Nov 02 18:33:55.116031 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:55.115998 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv"] Nov 02 18:33:55.116837 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:55.116801 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" podUID="cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" containerName="istio-proxy" containerID="cri-o://97f547e8a4ae6c6549a9de00ac8b2b2d6067394f0337c3688a5f9a663c0676a0" gracePeriod=30 Nov 02 18:33:58.203266 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.203223 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9"] Nov 02 18:33:58.458231 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.458042 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" containerID="cri-o://c2c54e0df990010f5261eee414ae00e9c8cc5a2187de64e70a377a9bc25f5337" gracePeriod=45 Nov 02 18:33:58.458449 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.458309 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" containerID="cri-o://07d61aadac87bf905d27d2818bc0cc1e8a1e92ed2bff73a00e77fe4729843537" gracePeriod=45 Nov 02 18:33:58.475733 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.475691 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" containerID="cri-o://38dd9bf500176a0efcfbc8e932252a6c050dd474ad7516ba3aa5f0dbb21af213" gracePeriod=45 Nov 02 18:33:58.777660 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.777633 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 18:33:58.783860 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.783835 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 18:33:58.793082 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.793061 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 18:33:58.902288 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.902253 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-model-cache\") pod \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " Nov 02 18:33:58.902476 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.902308 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-dshm\") pod \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " Nov 02 18:33:58.902476 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.902330 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-dshm\") pod \"bde15228-bd90-4a8a-a877-cac31762b285\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " Nov 02 18:33:58.902476 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.902361 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde15228-bd90-4a8a-a877-cac31762b285-kube-api-access-phflk\" (UniqueName: \"kubernetes.io/projected/bde15228-bd90-4a8a-a877-cac31762b285-kube-api-access-phflk\") pod \"bde15228-bd90-4a8a-a877-cac31762b285\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " Nov 02 18:33:58.902476 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.902407 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-tmp-dir\") pod \"bde15228-bd90-4a8a-a877-cac31762b285\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " Nov 02 18:33:58.902775 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.902580 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-model-cache" pod "a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" (UID: "a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:58.902775 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.902603 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76064488-6a4c-4f85-b1ee-75388968d2a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/76064488-6a4c-4f85-b1ee-75388968d2a1-tls-certs\") pod \"76064488-6a4c-4f85-b1ee-75388968d2a1\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " Nov 02 18:33:58.902775 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.902709 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-tmp-dir\") pod \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " Nov 02 18:33:58.902943 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.902787 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-model-cache\") pod \"bde15228-bd90-4a8a-a877-cac31762b285\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " Nov 02 18:33:58.902943 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.902826 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-home\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-home\") pod \"76064488-6a4c-4f85-b1ee-75388968d2a1\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " Nov 02 18:33:58.902943 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.902897 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-model-cache\") pod \"76064488-6a4c-4f85-b1ee-75388968d2a1\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " Nov 02 18:33:58.902943 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.902915 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-tmp-dir" pod "a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" (UID: "a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:58.902943 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.902940 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-dshm\") pod \"76064488-6a4c-4f85-b1ee-75388968d2a1\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " Nov 02 18:33:58.903228 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.902977 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76064488-6a4c-4f85-b1ee-75388968d2a1-kube-api-access-lrmtl\" (UniqueName: \"kubernetes.io/projected/76064488-6a4c-4f85-b1ee-75388968d2a1-kube-api-access-lrmtl\") pod \"76064488-6a4c-4f85-b1ee-75388968d2a1\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " Nov 02 18:33:58.903228 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.903008 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-home\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-home\") pod \"bde15228-bd90-4a8a-a877-cac31762b285\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " Nov 02 18:33:58.903228 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.903041 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-home\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-home\") pod \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " Nov 02 18:33:58.903228 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.903071 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-home" pod "76064488-6a4c-4f85-b1ee-75388968d2a1" (UID: "76064488-6a4c-4f85-b1ee-75388968d2a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:58.903228 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.903069 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-model-cache" pod "bde15228-bd90-4a8a-a877-cac31762b285" (UID: "bde15228-bd90-4a8a-a877-cac31762b285"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:58.903228 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.903102 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-tmp-dir\") pod \"76064488-6a4c-4f85-b1ee-75388968d2a1\" (UID: \"76064488-6a4c-4f85-b1ee-75388968d2a1\") " Nov 02 18:33:58.903228 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.903133 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-kube-api-access-6tjc7\" (UniqueName: \"kubernetes.io/projected/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-kube-api-access-6tjc7\") pod \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " Nov 02 18:33:58.903228 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.903164 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-tls-certs\") pod \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\" (UID: \"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff\") " Nov 02 18:33:58.903228 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.903173 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-model-cache" pod "76064488-6a4c-4f85-b1ee-75388968d2a1" (UID: "76064488-6a4c-4f85-b1ee-75388968d2a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:58.903754 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.903255 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bde15228-bd90-4a8a-a877-cac31762b285-tls-certs\" (UniqueName: \"kubernetes.io/secret/bde15228-bd90-4a8a-a877-cac31762b285-tls-certs\") pod \"bde15228-bd90-4a8a-a877-cac31762b285\" (UID: \"bde15228-bd90-4a8a-a877-cac31762b285\") " Nov 02 18:33:58.903940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.903828 5601 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-tmp-dir\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:58.904118 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.904092 5601 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-model-cache\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:58.904118 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.904100 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-tmp-dir" pod "76064488-6a4c-4f85-b1ee-75388968d2a1" (UID: "76064488-6a4c-4f85-b1ee-75388968d2a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:58.904118 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.904120 5601 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-model-cache\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:58.904118 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.904133 5601 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-model-cache\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:58.905028 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.904853 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-home" pod "bde15228-bd90-4a8a-a877-cac31762b285" (UID: "bde15228-bd90-4a8a-a877-cac31762b285"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:58.905028 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.904937 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-tmp-dir" pod "bde15228-bd90-4a8a-a877-cac31762b285" (UID: "bde15228-bd90-4a8a-a877-cac31762b285"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:58.905312 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.905236 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-home" pod "a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" (UID: "a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:58.906767 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.906710 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bde15228-bd90-4a8a-a877-cac31762b285-tls-certs" pod "bde15228-bd90-4a8a-a877-cac31762b285" (UID: "bde15228-bd90-4a8a-a877-cac31762b285"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:33:58.907046 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.906924 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-dshm" pod "76064488-6a4c-4f85-b1ee-75388968d2a1" (UID: "76064488-6a4c-4f85-b1ee-75388968d2a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:58.907046 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.906977 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76064488-6a4c-4f85-b1ee-75388968d2a1-tls-certs" pod "76064488-6a4c-4f85-b1ee-75388968d2a1" (UID: "76064488-6a4c-4f85-b1ee-75388968d2a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:33:58.907046 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.907001 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-dshm" pod "a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" (UID: "a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:58.908300 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.908263 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde15228-bd90-4a8a-a877-cac31762b285-kube-api-access-phflk" pod "bde15228-bd90-4a8a-a877-cac31762b285" (UID: "bde15228-bd90-4a8a-a877-cac31762b285"). InnerVolumeSpecName "kube-api-access-phflk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:58.908392 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.908317 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-dshm" pod "bde15228-bd90-4a8a-a877-cac31762b285" (UID: "bde15228-bd90-4a8a-a877-cac31762b285"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:33:58.908453 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.908394 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-kube-api-access-6tjc7" pod "a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" (UID: "a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff"). InnerVolumeSpecName "kube-api-access-6tjc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:58.908453 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.908405 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-tls-certs" pod "a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" (UID: "a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:33:58.908560 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:58.908543 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76064488-6a4c-4f85-b1ee-75388968d2a1-kube-api-access-lrmtl" pod "76064488-6a4c-4f85-b1ee-75388968d2a1" (UID: "76064488-6a4c-4f85-b1ee-75388968d2a1"). InnerVolumeSpecName "kube-api-access-lrmtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:59.005253 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.005161 5601 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-dshm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:59.005253 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.005195 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrmtl\" (UniqueName: \"kubernetes.io/projected/76064488-6a4c-4f85-b1ee-75388968d2a1-kube-api-access-lrmtl\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:59.005253 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.005204 5601 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-home\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:59.005253 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.005213 5601 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-home\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:59.005253 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.005221 5601 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-tmp-dir\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:59.005253 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.005230 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tjc7\" (UniqueName: \"kubernetes.io/projected/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-kube-api-access-6tjc7\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:59.005253 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.005238 5601 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-tls-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:59.005253 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.005246 5601 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bde15228-bd90-4a8a-a877-cac31762b285-tls-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:59.005253 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.005253 5601 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff-dshm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:59.005253 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.005261 5601 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-dshm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:59.005781 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.005269 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phflk\" (UniqueName: \"kubernetes.io/projected/bde15228-bd90-4a8a-a877-cac31762b285-kube-api-access-phflk\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:59.005781 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.005279 5601 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bde15228-bd90-4a8a-a877-cac31762b285-tmp-dir\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:59.005781 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.005289 5601 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76064488-6a4c-4f85-b1ee-75388968d2a1-tls-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:59.005781 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.005296 5601 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76064488-6a4c-4f85-b1ee-75388968d2a1-home\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:33:59.564685 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.564642 5601 generic.go:356] "Generic (PLEG): container finished" podID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerID="07d61aadac87bf905d27d2818bc0cc1e8a1e92ed2bff73a00e77fe4729843537" exitCode=0 Nov 02 18:33:59.564685 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.564668 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" Nov 02 18:33:59.565230 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.564722 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" event={"ID":"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff","Type":"ContainerDied","Data":"07d61aadac87bf905d27d2818bc0cc1e8a1e92ed2bff73a00e77fe4729843537"} Nov 02 18:33:59.565230 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.564768 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl" event={"ID":"a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff","Type":"ContainerDied","Data":"94bad7d2400b886fcfc124c0acd15d151b93daa002dc3bda50bda6cd45f8f7f4"} Nov 02 18:33:59.565230 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.564799 5601 scope.go:122] "RemoveContainer" containerID="07d61aadac87bf905d27d2818bc0cc1e8a1e92ed2bff73a00e77fe4729843537" Nov 02 18:33:59.566323 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.566269 5601 generic.go:356] "Generic (PLEG): container finished" podID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerID="c2c54e0df990010f5261eee414ae00e9c8cc5a2187de64e70a377a9bc25f5337" exitCode=0 Nov 02 18:33:59.566435 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.566307 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" Nov 02 18:33:59.566435 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.566328 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" event={"ID":"76064488-6a4c-4f85-b1ee-75388968d2a1","Type":"ContainerDied","Data":"c2c54e0df990010f5261eee414ae00e9c8cc5a2187de64e70a377a9bc25f5337"} Nov 02 18:33:59.566435 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.566363 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz" event={"ID":"76064488-6a4c-4f85-b1ee-75388968d2a1","Type":"ContainerDied","Data":"4612491f854588242fc420eecf966240283fd4ed678598d45b55bf5b044537f2"} Nov 02 18:33:59.567854 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.567833 5601 generic.go:356] "Generic (PLEG): container finished" podID="bde15228-bd90-4a8a-a877-cac31762b285" containerID="38dd9bf500176a0efcfbc8e932252a6c050dd474ad7516ba3aa5f0dbb21af213" exitCode=0 Nov 02 18:33:59.567990 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.567867 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" Nov 02 18:33:59.567990 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.567867 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" event={"ID":"bde15228-bd90-4a8a-a877-cac31762b285","Type":"ContainerDied","Data":"38dd9bf500176a0efcfbc8e932252a6c050dd474ad7516ba3aa5f0dbb21af213"} Nov 02 18:33:59.568133 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.567993 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5" event={"ID":"bde15228-bd90-4a8a-a877-cac31762b285","Type":"ContainerDied","Data":"24042b8ec8d15b2071e572a1118c18ec9550f93de8abd11535b5f37384c7baa2"} Nov 02 18:33:59.576032 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.576002 5601 scope.go:122] "RemoveContainer" containerID="07d61aadac87bf905d27d2818bc0cc1e8a1e92ed2bff73a00e77fe4729843537" Nov 02 18:33:59.576342 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:33:59.576323 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d61aadac87bf905d27d2818bc0cc1e8a1e92ed2bff73a00e77fe4729843537\": container with ID starting with 07d61aadac87bf905d27d2818bc0cc1e8a1e92ed2bff73a00e77fe4729843537 not found: ID does not exist" containerID="07d61aadac87bf905d27d2818bc0cc1e8a1e92ed2bff73a00e77fe4729843537" Nov 02 18:33:59.576408 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.576351 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d61aadac87bf905d27d2818bc0cc1e8a1e92ed2bff73a00e77fe4729843537"} err="failed to get container status \"07d61aadac87bf905d27d2818bc0cc1e8a1e92ed2bff73a00e77fe4729843537\": rpc error: code = NotFound desc = could not find container \"07d61aadac87bf905d27d2818bc0cc1e8a1e92ed2bff73a00e77fe4729843537\": container with ID starting with 07d61aadac87bf905d27d2818bc0cc1e8a1e92ed2bff73a00e77fe4729843537 not found: ID does not exist" Nov 02 18:33:59.576408 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.576368 5601 scope.go:122] "RemoveContainer" containerID="c2c54e0df990010f5261eee414ae00e9c8cc5a2187de64e70a377a9bc25f5337" Nov 02 18:33:59.586196 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.586169 5601 scope.go:122] "RemoveContainer" containerID="c2c54e0df990010f5261eee414ae00e9c8cc5a2187de64e70a377a9bc25f5337" Nov 02 18:33:59.586490 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:33:59.586469 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c54e0df990010f5261eee414ae00e9c8cc5a2187de64e70a377a9bc25f5337\": container with ID starting with c2c54e0df990010f5261eee414ae00e9c8cc5a2187de64e70a377a9bc25f5337 not found: ID does not exist" containerID="c2c54e0df990010f5261eee414ae00e9c8cc5a2187de64e70a377a9bc25f5337" Nov 02 18:33:59.586548 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.586498 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c54e0df990010f5261eee414ae00e9c8cc5a2187de64e70a377a9bc25f5337"} err="failed to get container status \"c2c54e0df990010f5261eee414ae00e9c8cc5a2187de64e70a377a9bc25f5337\": rpc error: code = NotFound desc = could not find container \"c2c54e0df990010f5261eee414ae00e9c8cc5a2187de64e70a377a9bc25f5337\": container with ID starting with c2c54e0df990010f5261eee414ae00e9c8cc5a2187de64e70a377a9bc25f5337 not found: ID does not exist" Nov 02 18:33:59.586548 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.586515 5601 scope.go:122] "RemoveContainer" containerID="38dd9bf500176a0efcfbc8e932252a6c050dd474ad7516ba3aa5f0dbb21af213" Nov 02 18:33:59.597164 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.597133 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5"] Nov 02 18:33:59.598052 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.598028 5601 scope.go:122] "RemoveContainer" containerID="38dd9bf500176a0efcfbc8e932252a6c050dd474ad7516ba3aa5f0dbb21af213" Nov 02 18:33:59.598408 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:33:59.598378 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38dd9bf500176a0efcfbc8e932252a6c050dd474ad7516ba3aa5f0dbb21af213\": container with ID starting with 38dd9bf500176a0efcfbc8e932252a6c050dd474ad7516ba3aa5f0dbb21af213 not found: ID does not exist" containerID="38dd9bf500176a0efcfbc8e932252a6c050dd474ad7516ba3aa5f0dbb21af213" Nov 02 18:33:59.598528 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.598409 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38dd9bf500176a0efcfbc8e932252a6c050dd474ad7516ba3aa5f0dbb21af213"} err="failed to get container status \"38dd9bf500176a0efcfbc8e932252a6c050dd474ad7516ba3aa5f0dbb21af213\": rpc error: code = NotFound desc = could not find container \"38dd9bf500176a0efcfbc8e932252a6c050dd474ad7516ba3aa5f0dbb21af213\": container with ID starting with 38dd9bf500176a0efcfbc8e932252a6c050dd474ad7516ba3aa5f0dbb21af213 not found: ID does not exist" Nov 02 18:33:59.599630 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.599595 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/simulator-w0-4b97c6-kserve-5997bf6fc8-kmjl5"] Nov 02 18:33:59.608097 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.608063 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl"] Nov 02 18:33:59.614470 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.614435 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/distinct-2-w0-4b97c6-kserve-7bc8db6c8f-lrtcl"] Nov 02 18:33:59.623965 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.623933 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz"] Nov 02 18:33:59.629173 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:33:59.629144 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4b97c6/distinct-w0-4b97c6-kserve-6c8454bc5d-nz5mz"] Nov 02 18:34:00.404895 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.404816 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:34:00.522110 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.522063 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-token\" (UniqueName: \"kubernetes.io/projected/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-token\") pod \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " Nov 02 18:34:00.522277 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.522123 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-envoy\") pod \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " Nov 02 18:34:00.522277 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.522147 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-podinfo\") pod \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " Nov 02 18:34:00.522277 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.522191 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-data\") pod \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " Nov 02 18:34:00.522277 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.522213 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-kube-api-access-vbh6f\" (UniqueName: \"kubernetes.io/projected/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-kube-api-access-vbh6f\") pod \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " Nov 02 18:34:00.522277 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.522248 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-workload-socket\") pod \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " Nov 02 18:34:00.522582 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.522345 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-workload-certs\") pod \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " Nov 02 18:34:00.522582 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.522379 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istiod-ca-cert\") pod \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " Nov 02 18:34:00.522582 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.522474 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-credential-socket\") pod \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\" (UID: \"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a\") " Nov 02 18:34:00.522789 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.522561 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-data" pod "cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" (UID: "cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:00.522856 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.522808 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-credential-socket" pod "cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" (UID: "cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:00.522856 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.522799 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-workload-certs" pod "cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" (UID: "cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:00.523032 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.522960 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-workload-socket" pod "cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" (UID: "cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:00.523159 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.523047 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:00.523159 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.523064 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:00.523159 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.523077 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:00.523159 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.523119 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istiod-ca-cert" pod "cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" (UID: "cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:00.524739 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.524710 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-envoy" pod "cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" (UID: "cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:00.524843 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.524715 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-kube-api-access-vbh6f" pod "cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" (UID: "cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a"). InnerVolumeSpecName "kube-api-access-vbh6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:00.524989 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.524967 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-token" pod "cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" (UID: "cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:00.525098 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.524974 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-podinfo" pod "cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" (UID: "cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:34:00.576958 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.576918 5601 generic.go:356] "Generic (PLEG): container finished" podID="cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" containerID="97f547e8a4ae6c6549a9de00ac8b2b2d6067394f0337c3688a5f9a663c0676a0" exitCode=0 Nov 02 18:34:00.577440 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.576977 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" Nov 02 18:34:00.577440 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.577006 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" event={"ID":"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a","Type":"ContainerDied","Data":"97f547e8a4ae6c6549a9de00ac8b2b2d6067394f0337c3688a5f9a663c0676a0"} Nov 02 18:34:00.577440 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.577058 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv" event={"ID":"cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a","Type":"ContainerDied","Data":"680edc99eb18c2a5117e0e07400686b7ec585151e46e8dc850299b38a280f754"} Nov 02 18:34:00.577440 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.577080 5601 scope.go:122] "RemoveContainer" containerID="97f547e8a4ae6c6549a9de00ac8b2b2d6067394f0337c3688a5f9a663c0676a0" Nov 02 18:34:00.587191 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.587169 5601 scope.go:122] "RemoveContainer" containerID="97f547e8a4ae6c6549a9de00ac8b2b2d6067394f0337c3688a5f9a663c0676a0" Nov 02 18:34:00.587467 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:34:00.587444 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97f547e8a4ae6c6549a9de00ac8b2b2d6067394f0337c3688a5f9a663c0676a0\": container with ID starting with 97f547e8a4ae6c6549a9de00ac8b2b2d6067394f0337c3688a5f9a663c0676a0 not found: ID does not exist" containerID="97f547e8a4ae6c6549a9de00ac8b2b2d6067394f0337c3688a5f9a663c0676a0" Nov 02 18:34:00.587519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.587475 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97f547e8a4ae6c6549a9de00ac8b2b2d6067394f0337c3688a5f9a663c0676a0"} err="failed to get container status \"97f547e8a4ae6c6549a9de00ac8b2b2d6067394f0337c3688a5f9a663c0676a0\": rpc error: code = NotFound desc = could not find container \"97f547e8a4ae6c6549a9de00ac8b2b2d6067394f0337c3688a5f9a663c0676a0\": container with ID starting with 97f547e8a4ae6c6549a9de00ac8b2b2d6067394f0337c3688a5f9a663c0676a0 not found: ID does not exist" Nov 02 18:34:00.600125 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.600096 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv"] Nov 02 18:34:00.603415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.603388 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-4n9pv"] Nov 02 18:34:00.624528 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.624493 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:00.624528 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.624518 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:00.624753 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.624534 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:00.624753 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.624550 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbh6f\" (UniqueName: \"kubernetes.io/projected/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-kube-api-access-vbh6f\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:00.624753 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.624564 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:00.624753 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:00.624573 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:01.425184 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:01.425147 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" path="/var/lib/kubelet/pods/76064488-6a4c-4f85-b1ee-75388968d2a1/volumes" Nov 02 18:34:01.425583 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:01.425569 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" path="/var/lib/kubelet/pods/a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff/volumes" Nov 02 18:34:01.425999 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:01.425985 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bde15228-bd90-4a8a-a877-cac31762b285" path="/var/lib/kubelet/pods/bde15228-bd90-4a8a-a877-cac31762b285/volumes" Nov 02 18:34:01.426708 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:01.426692 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" path="/var/lib/kubelet/pods/cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a/volumes" Nov 02 18:34:03.203717 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:03.203665 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" podUID="adb37323-1f64-44d2-906c-853342775292" containerName="payload-pre-processing" containerID="cri-o://01222030ce3bf1c9c2b872752405066005ffbee16ff02683e7a48bc02087cc73" gracePeriod=25 Nov 02 18:34:04.484884 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:34:04.484849 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:34:12.130256 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.130213 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7"] Nov 02 18:34:12.130773 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.130484 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" podUID="c2852736-70fa-4810-9f5a-6211fae2d2c9" containerName="maas-api" containerID="cri-o://28abdcb758a94259b7dd8b55ed54ba2bb71ee3b09b67747c9b33e10ee19b3a54" gracePeriod=30 Nov 02 18:34:12.444419 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.444386 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" Nov 02 18:34:12.557089 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.557044 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2852736-70fa-4810-9f5a-6211fae2d2c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2852736-70fa-4810-9f5a-6211fae2d2c9-openshift-service-ca\") pod \"c2852736-70fa-4810-9f5a-6211fae2d2c9\" (UID: \"c2852736-70fa-4810-9f5a-6211fae2d2c9\") " Nov 02 18:34:12.557290 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.557109 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2852736-70fa-4810-9f5a-6211fae2d2c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2852736-70fa-4810-9f5a-6211fae2d2c9-metrics-certs\") pod \"c2852736-70fa-4810-9f5a-6211fae2d2c9\" (UID: \"c2852736-70fa-4810-9f5a-6211fae2d2c9\") " Nov 02 18:34:12.557337 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.557304 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2852736-70fa-4810-9f5a-6211fae2d2c9-kube-api-access-jxbwt\" (UniqueName: \"kubernetes.io/projected/c2852736-70fa-4810-9f5a-6211fae2d2c9-kube-api-access-jxbwt\") pod \"c2852736-70fa-4810-9f5a-6211fae2d2c9\" (UID: \"c2852736-70fa-4810-9f5a-6211fae2d2c9\") " Nov 02 18:34:12.557384 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.557356 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2852736-70fa-4810-9f5a-6211fae2d2c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2852736-70fa-4810-9f5a-6211fae2d2c9-maas-api-tls\") pod \"c2852736-70fa-4810-9f5a-6211fae2d2c9\" (UID: \"c2852736-70fa-4810-9f5a-6211fae2d2c9\") " Nov 02 18:34:12.557427 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.557397 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2852736-70fa-4810-9f5a-6211fae2d2c9-openshift-service-ca" pod "c2852736-70fa-4810-9f5a-6211fae2d2c9" (UID: "c2852736-70fa-4810-9f5a-6211fae2d2c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:12.557754 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.557735 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2852736-70fa-4810-9f5a-6211fae2d2c9-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:12.559452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.559421 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2852736-70fa-4810-9f5a-6211fae2d2c9-metrics-certs" pod "c2852736-70fa-4810-9f5a-6211fae2d2c9" (UID: "c2852736-70fa-4810-9f5a-6211fae2d2c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:34:12.559452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.559438 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2852736-70fa-4810-9f5a-6211fae2d2c9-maas-api-tls" pod "c2852736-70fa-4810-9f5a-6211fae2d2c9" (UID: "c2852736-70fa-4810-9f5a-6211fae2d2c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:34:12.559587 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.559441 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2852736-70fa-4810-9f5a-6211fae2d2c9-kube-api-access-jxbwt" pod "c2852736-70fa-4810-9f5a-6211fae2d2c9" (UID: "c2852736-70fa-4810-9f5a-6211fae2d2c9"). InnerVolumeSpecName "kube-api-access-jxbwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:12.649702 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.649666 5601 generic.go:356] "Generic (PLEG): container finished" podID="c2852736-70fa-4810-9f5a-6211fae2d2c9" containerID="28abdcb758a94259b7dd8b55ed54ba2bb71ee3b09b67747c9b33e10ee19b3a54" exitCode=0 Nov 02 18:34:12.649905 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.649718 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" Nov 02 18:34:12.649905 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.649752 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" event={"ID":"c2852736-70fa-4810-9f5a-6211fae2d2c9","Type":"ContainerDied","Data":"28abdcb758a94259b7dd8b55ed54ba2bb71ee3b09b67747c9b33e10ee19b3a54"} Nov 02 18:34:12.649905 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.649807 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7" event={"ID":"c2852736-70fa-4810-9f5a-6211fae2d2c9","Type":"ContainerDied","Data":"4b5f6b890406e42d6218e44b9ee49b300b8ec63b2147e548872e3b8f86b5ae9b"} Nov 02 18:34:12.649905 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.649831 5601 scope.go:122] "RemoveContainer" containerID="28abdcb758a94259b7dd8b55ed54ba2bb71ee3b09b67747c9b33e10ee19b3a54" Nov 02 18:34:12.658499 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.658471 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxbwt\" (UniqueName: \"kubernetes.io/projected/c2852736-70fa-4810-9f5a-6211fae2d2c9-kube-api-access-jxbwt\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:12.658690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.658503 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2852736-70fa-4810-9f5a-6211fae2d2c9-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:12.658690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.658519 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2852736-70fa-4810-9f5a-6211fae2d2c9-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:12.661054 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.661025 5601 scope.go:122] "RemoveContainer" containerID="28abdcb758a94259b7dd8b55ed54ba2bb71ee3b09b67747c9b33e10ee19b3a54" Nov 02 18:34:12.661385 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:34:12.661363 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28abdcb758a94259b7dd8b55ed54ba2bb71ee3b09b67747c9b33e10ee19b3a54\": container with ID starting with 28abdcb758a94259b7dd8b55ed54ba2bb71ee3b09b67747c9b33e10ee19b3a54 not found: ID does not exist" containerID="28abdcb758a94259b7dd8b55ed54ba2bb71ee3b09b67747c9b33e10ee19b3a54" Nov 02 18:34:12.661476 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.661398 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28abdcb758a94259b7dd8b55ed54ba2bb71ee3b09b67747c9b33e10ee19b3a54"} err="failed to get container status \"28abdcb758a94259b7dd8b55ed54ba2bb71ee3b09b67747c9b33e10ee19b3a54\": rpc error: code = NotFound desc = could not find container \"28abdcb758a94259b7dd8b55ed54ba2bb71ee3b09b67747c9b33e10ee19b3a54\": container with ID starting with 28abdcb758a94259b7dd8b55ed54ba2bb71ee3b09b67747c9b33e10ee19b3a54 not found: ID does not exist" Nov 02 18:34:12.672822 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.672739 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7"] Nov 02 18:34:12.678234 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:12.678198 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b97c6-6d89bcfd5b-9hcx7"] Nov 02 18:34:13.098202 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:13.098105 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" podUID="adb37323-1f64-44d2-906c-853342775292" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:34:13.426050 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:13.426005 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2852736-70fa-4810-9f5a-6211fae2d2c9" path="/var/lib/kubelet/pods/c2852736-70fa-4810-9f5a-6211fae2d2c9/volumes" Nov 02 18:34:20.881538 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:20.881276 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz"] Nov 02 18:34:20.882180 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:20.881530 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" podUID="db8e57a9-62e6-4229-921d-f5a40cde2a7a" containerName="istio-proxy" containerID="cri-o://da170cbd1d20a185d4a421951af721c985ed5b9830b26e53b9d39c9795f15b82" gracePeriod=30 Nov 02 18:34:23.098256 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:23.098205 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" podUID="adb37323-1f64-44d2-906c-853342775292" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:34:23.464983 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:23.464926 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" Nov 02 18:34:23.597411 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:23.597375 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adb37323-1f64-44d2-906c-853342775292-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb37323-1f64-44d2-906c-853342775292-plugins-config-volume\") pod \"adb37323-1f64-44d2-906c-853342775292\" (UID: \"adb37323-1f64-44d2-906c-853342775292\") " Nov 02 18:34:23.597678 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:23.597495 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb37323-1f64-44d2-906c-853342775292-kube-api-access-r6fn8\" (UniqueName: \"kubernetes.io/projected/adb37323-1f64-44d2-906c-853342775292-kube-api-access-r6fn8\") pod \"adb37323-1f64-44d2-906c-853342775292\" (UID: \"adb37323-1f64-44d2-906c-853342775292\") " Nov 02 18:34:23.597894 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:23.597866 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb37323-1f64-44d2-906c-853342775292-plugins-config-volume" pod "adb37323-1f64-44d2-906c-853342775292" (UID: "adb37323-1f64-44d2-906c-853342775292"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:23.599784 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:23.599760 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb37323-1f64-44d2-906c-853342775292-kube-api-access-r6fn8" pod "adb37323-1f64-44d2-906c-853342775292" (UID: "adb37323-1f64-44d2-906c-853342775292"). InnerVolumeSpecName "kube-api-access-r6fn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:23.699220 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:23.699171 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6fn8\" (UniqueName: \"kubernetes.io/projected/adb37323-1f64-44d2-906c-853342775292-kube-api-access-r6fn8\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:23.699220 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:23.699210 5601 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb37323-1f64-44d2-906c-853342775292-plugins-config-volume\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:23.715327 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:23.715286 5601 generic.go:356] "Generic (PLEG): container finished" podID="adb37323-1f64-44d2-906c-853342775292" containerID="01222030ce3bf1c9c2b872752405066005ffbee16ff02683e7a48bc02087cc73" exitCode=0 Nov 02 18:34:23.715512 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:23.715335 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" Nov 02 18:34:23.715512 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:23.715364 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" event={"ID":"adb37323-1f64-44d2-906c-853342775292","Type":"ContainerDied","Data":"01222030ce3bf1c9c2b872752405066005ffbee16ff02683e7a48bc02087cc73"} Nov 02 18:34:23.715512 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:23.715417 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9" event={"ID":"adb37323-1f64-44d2-906c-853342775292","Type":"ContainerDied","Data":"b6e0078e88d79e680489f784e427d79b3d742d112e781eeb0ceb87fd407e6d64"} Nov 02 18:34:23.715512 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:23.715442 5601 scope.go:122] "RemoveContainer" containerID="01222030ce3bf1c9c2b872752405066005ffbee16ff02683e7a48bc02087cc73" Nov 02 18:34:23.728671 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:23.728648 5601 scope.go:122] "RemoveContainer" containerID="01222030ce3bf1c9c2b872752405066005ffbee16ff02683e7a48bc02087cc73" Nov 02 18:34:23.728943 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:34:23.728923 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01222030ce3bf1c9c2b872752405066005ffbee16ff02683e7a48bc02087cc73\": container with ID starting with 01222030ce3bf1c9c2b872752405066005ffbee16ff02683e7a48bc02087cc73 not found: ID does not exist" containerID="01222030ce3bf1c9c2b872752405066005ffbee16ff02683e7a48bc02087cc73" Nov 02 18:34:23.728997 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:23.728953 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01222030ce3bf1c9c2b872752405066005ffbee16ff02683e7a48bc02087cc73"} err="failed to get container status \"01222030ce3bf1c9c2b872752405066005ffbee16ff02683e7a48bc02087cc73\": rpc error: code = NotFound desc = could not find container \"01222030ce3bf1c9c2b872752405066005ffbee16ff02683e7a48bc02087cc73\": container with ID starting with 01222030ce3bf1c9c2b872752405066005ffbee16ff02683e7a48bc02087cc73 not found: ID does not exist" Nov 02 18:34:23.754340 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:23.754277 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9"] Nov 02 18:34:23.757003 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:23.756971 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4b97c6-67955696c5-wfzv9"] Nov 02 18:34:24.618858 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:24.611608 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn"] Nov 02 18:34:24.618858 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:24.611942 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" podUID="48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" containerName="istio-proxy" containerID="cri-o://d59a2c5453a2561c58293a1dd802fcddffddf3c07bfe2bfbfe78b17a7644e462" gracePeriod=30 Nov 02 18:34:25.425693 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:25.425663 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adb37323-1f64-44d2-906c-853342775292" path="/var/lib/kubelet/pods/adb37323-1f64-44d2-906c-853342775292/volumes" Nov 02 18:34:26.157299 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.157231 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 18:34:26.229304 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.229262 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istiod-ca-cert\") pod \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " Nov 02 18:34:26.229491 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.229355 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-credential-socket\") pod \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " Nov 02 18:34:26.229491 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.229394 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-workload-certs\") pod \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " Nov 02 18:34:26.229491 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.229448 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-token\" (UniqueName: \"kubernetes.io/projected/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-token\") pod \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " Nov 02 18:34:26.229740 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.229518 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db8e57a9-62e6-4229-921d-f5a40cde2a7a-kube-api-access-j9pvk\" (UniqueName: \"kubernetes.io/projected/db8e57a9-62e6-4229-921d-f5a40cde2a7a-kube-api-access-j9pvk\") pod \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " Nov 02 18:34:26.229740 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.229565 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-data\") pod \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " Nov 02 18:34:26.229740 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.229606 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-podinfo\") pod \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " Nov 02 18:34:26.229916 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.229784 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-workload-certs" pod "db8e57a9-62e6-4229-921d-f5a40cde2a7a" (UID: "db8e57a9-62e6-4229-921d-f5a40cde2a7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:26.229916 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.229800 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-envoy\") pod \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " Nov 02 18:34:26.229916 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.229838 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-workload-socket\") pod \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\" (UID: \"db8e57a9-62e6-4229-921d-f5a40cde2a7a\") " Nov 02 18:34:26.229916 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.229704 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istiod-ca-cert" pod "db8e57a9-62e6-4229-921d-f5a40cde2a7a" (UID: "db8e57a9-62e6-4229-921d-f5a40cde2a7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:26.230142 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.229962 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-data" pod "db8e57a9-62e6-4229-921d-f5a40cde2a7a" (UID: "db8e57a9-62e6-4229-921d-f5a40cde2a7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:26.230142 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.230025 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-credential-socket" pod "db8e57a9-62e6-4229-921d-f5a40cde2a7a" (UID: "db8e57a9-62e6-4229-921d-f5a40cde2a7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:26.230271 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.230150 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-workload-socket" pod "db8e57a9-62e6-4229-921d-f5a40cde2a7a" (UID: "db8e57a9-62e6-4229-921d-f5a40cde2a7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:26.230271 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.230249 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:26.230378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.230271 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:26.230378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.230286 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:26.230378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.230302 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:26.230378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.230317 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:26.231957 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.231927 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-envoy" pod "db8e57a9-62e6-4229-921d-f5a40cde2a7a" (UID: "db8e57a9-62e6-4229-921d-f5a40cde2a7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:26.232058 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.231990 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8e57a9-62e6-4229-921d-f5a40cde2a7a-kube-api-access-j9pvk" pod "db8e57a9-62e6-4229-921d-f5a40cde2a7a" (UID: "db8e57a9-62e6-4229-921d-f5a40cde2a7a"). InnerVolumeSpecName "kube-api-access-j9pvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:26.232104 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.232058 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-token" pod "db8e57a9-62e6-4229-921d-f5a40cde2a7a" (UID: "db8e57a9-62e6-4229-921d-f5a40cde2a7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:26.232215 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.232193 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-podinfo" pod "db8e57a9-62e6-4229-921d-f5a40cde2a7a" (UID: "db8e57a9-62e6-4229-921d-f5a40cde2a7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:34:26.331705 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.331669 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:26.331705 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.331701 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9pvk\" (UniqueName: \"kubernetes.io/projected/db8e57a9-62e6-4229-921d-f5a40cde2a7a-kube-api-access-j9pvk\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:26.331705 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.331711 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:26.331947 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.331720 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8e57a9-62e6-4229-921d-f5a40cde2a7a-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:26.742069 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.742030 5601 generic.go:356] "Generic (PLEG): container finished" podID="db8e57a9-62e6-4229-921d-f5a40cde2a7a" containerID="da170cbd1d20a185d4a421951af721c985ed5b9830b26e53b9d39c9795f15b82" exitCode=0 Nov 02 18:34:26.742268 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.742086 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" Nov 02 18:34:26.742268 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.742104 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" event={"ID":"db8e57a9-62e6-4229-921d-f5a40cde2a7a","Type":"ContainerDied","Data":"da170cbd1d20a185d4a421951af721c985ed5b9830b26e53b9d39c9795f15b82"} Nov 02 18:34:26.742268 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.742147 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz" event={"ID":"db8e57a9-62e6-4229-921d-f5a40cde2a7a","Type":"ContainerDied","Data":"1f23bb7315cb0832f8e76d4d1276dcc6f86f755c7d1064cef8a8bbad182b8999"} Nov 02 18:34:26.742268 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.742170 5601 scope.go:122] "RemoveContainer" containerID="da170cbd1d20a185d4a421951af721c985ed5b9830b26e53b9d39c9795f15b82" Nov 02 18:34:26.752112 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.752087 5601 scope.go:122] "RemoveContainer" containerID="e3816cb714ea3f3f23240adae24bafd276db6a0a51b6df93dd5432bf1b7d1981" Nov 02 18:34:26.761219 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.761200 5601 scope.go:122] "RemoveContainer" containerID="da170cbd1d20a185d4a421951af721c985ed5b9830b26e53b9d39c9795f15b82" Nov 02 18:34:26.761480 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:34:26.761459 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da170cbd1d20a185d4a421951af721c985ed5b9830b26e53b9d39c9795f15b82\": container with ID starting with da170cbd1d20a185d4a421951af721c985ed5b9830b26e53b9d39c9795f15b82 not found: ID does not exist" containerID="da170cbd1d20a185d4a421951af721c985ed5b9830b26e53b9d39c9795f15b82" Nov 02 18:34:26.761523 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.761494 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da170cbd1d20a185d4a421951af721c985ed5b9830b26e53b9d39c9795f15b82"} err="failed to get container status \"da170cbd1d20a185d4a421951af721c985ed5b9830b26e53b9d39c9795f15b82\": rpc error: code = NotFound desc = could not find container \"da170cbd1d20a185d4a421951af721c985ed5b9830b26e53b9d39c9795f15b82\": container with ID starting with da170cbd1d20a185d4a421951af721c985ed5b9830b26e53b9d39c9795f15b82 not found: ID does not exist" Nov 02 18:34:26.761523 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.761519 5601 scope.go:122] "RemoveContainer" containerID="e3816cb714ea3f3f23240adae24bafd276db6a0a51b6df93dd5432bf1b7d1981" Nov 02 18:34:26.761871 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:34:26.761829 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3816cb714ea3f3f23240adae24bafd276db6a0a51b6df93dd5432bf1b7d1981\": container with ID starting with e3816cb714ea3f3f23240adae24bafd276db6a0a51b6df93dd5432bf1b7d1981 not found: ID does not exist" containerID="e3816cb714ea3f3f23240adae24bafd276db6a0a51b6df93dd5432bf1b7d1981" Nov 02 18:34:26.761871 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.761862 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3816cb714ea3f3f23240adae24bafd276db6a0a51b6df93dd5432bf1b7d1981"} err="failed to get container status \"e3816cb714ea3f3f23240adae24bafd276db6a0a51b6df93dd5432bf1b7d1981\": rpc error: code = NotFound desc = could not find container \"e3816cb714ea3f3f23240adae24bafd276db6a0a51b6df93dd5432bf1b7d1981\": container with ID starting with e3816cb714ea3f3f23240adae24bafd276db6a0a51b6df93dd5432bf1b7d1981 not found: ID does not exist" Nov 02 18:34:26.765419 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.765392 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz"] Nov 02 18:34:26.767204 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:26.767183 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-5p2hz"] Nov 02 18:34:27.424953 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:27.424917 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db8e57a9-62e6-4229-921d-f5a40cde2a7a" path="/var/lib/kubelet/pods/db8e57a9-62e6-4229-921d-f5a40cde2a7a/volumes" Nov 02 18:34:27.815311 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:27.815219 5601 patch_prober.go:29] interesting pod/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:34:27.815311 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:27.815283 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" podUID="48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:34:29.772652 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.772590 5601 generic.go:356] "Generic (PLEG): container finished" podID="48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" containerID="d59a2c5453a2561c58293a1dd802fcddffddf3c07bfe2bfbfe78b17a7644e462" exitCode=0 Nov 02 18:34:29.773065 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.772654 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" event={"ID":"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4","Type":"ContainerDied","Data":"d59a2c5453a2561c58293a1dd802fcddffddf3c07bfe2bfbfe78b17a7644e462"} Nov 02 18:34:29.887338 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.887270 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:34:29.969461 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.969427 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-workload-certs\") pod \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " Nov 02 18:34:29.969682 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.969510 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-podinfo\") pod \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " Nov 02 18:34:29.969682 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.969557 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-credential-socket\") pod \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " Nov 02 18:34:29.969682 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.969593 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-kube-api-access-772v6\" (UniqueName: \"kubernetes.io/projected/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-kube-api-access-772v6\") pod \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " Nov 02 18:34:29.969682 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.969673 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-workload-socket\") pod \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " Nov 02 18:34:29.969906 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.969708 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istiod-ca-cert\") pod \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " Nov 02 18:34:29.969906 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.969805 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-workload-certs" pod "48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" (UID: "48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:29.969906 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.969849 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-envoy\") pod \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " Nov 02 18:34:29.969906 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.969862 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-credential-socket" pod "48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" (UID: "48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:29.970127 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.969922 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-data\") pod \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " Nov 02 18:34:29.970127 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.969961 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-token\" (UniqueName: \"kubernetes.io/projected/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-token\") pod \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\" (UID: \"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4\") " Nov 02 18:34:29.970127 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.970071 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-workload-socket" pod "48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" (UID: "48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:29.970295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.970207 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-data" pod "48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" (UID: "48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:29.970295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.970246 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istiod-ca-cert" pod "48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" (UID: "48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:29.970417 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.970394 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:29.970417 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.970413 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:29.970536 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.970426 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:29.970536 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.970440 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:29.970536 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.970454 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:29.971932 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.971907 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-podinfo" pod "48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" (UID: "48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:34:29.972066 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.972044 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-kube-api-access-772v6" pod "48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" (UID: "48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4"). InnerVolumeSpecName "kube-api-access-772v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:29.972130 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.972111 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-envoy" pod "48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" (UID: "48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:29.972257 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:29.972239 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-token" pod "48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" (UID: "48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:30.071247 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:30.071208 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-772v6\" (UniqueName: \"kubernetes.io/projected/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-kube-api-access-772v6\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:30.071247 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:30.071242 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:30.071247 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:30.071254 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:30.071512 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:30.071262 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:30.779307 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:30.779269 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" Nov 02 18:34:30.779784 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:30.779274 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn" event={"ID":"48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4","Type":"ContainerDied","Data":"e728da02929724a2b9b6370afe779447132df98b929c749f44aec0af7542b7c2"} Nov 02 18:34:30.779784 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:30.779410 5601 scope.go:122] "RemoveContainer" containerID="d59a2c5453a2561c58293a1dd802fcddffddf3c07bfe2bfbfe78b17a7644e462" Nov 02 18:34:30.805782 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:30.805747 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn"] Nov 02 18:34:30.807515 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:30.807490 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-ljkzn"] Nov 02 18:34:31.425521 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:31.425482 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" path="/var/lib/kubelet/pods/48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4/volumes" Nov 02 18:34:32.119730 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.119537 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22"] Nov 02 18:34:32.120259 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.119821 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" podUID="d8058f31-67f2-4f71-8f17-5f839c042a2f" containerName="maas-api" containerID="cri-o://cd3ed2b7a2cb6d34a01d609617839afb6a08ef7f4edb83d457ddc6f6c93f734b" gracePeriod=30 Nov 02 18:34:32.415784 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.415753 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" Nov 02 18:34:32.497157 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.497122 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8058f31-67f2-4f71-8f17-5f839c042a2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8058f31-67f2-4f71-8f17-5f839c042a2f-maas-api-tls\") pod \"d8058f31-67f2-4f71-8f17-5f839c042a2f\" (UID: \"d8058f31-67f2-4f71-8f17-5f839c042a2f\") " Nov 02 18:34:32.497383 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.497164 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8058f31-67f2-4f71-8f17-5f839c042a2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8058f31-67f2-4f71-8f17-5f839c042a2f-openshift-service-ca\") pod \"d8058f31-67f2-4f71-8f17-5f839c042a2f\" (UID: \"d8058f31-67f2-4f71-8f17-5f839c042a2f\") " Nov 02 18:34:32.497383 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.497227 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8058f31-67f2-4f71-8f17-5f839c042a2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8058f31-67f2-4f71-8f17-5f839c042a2f-metrics-certs\") pod \"d8058f31-67f2-4f71-8f17-5f839c042a2f\" (UID: \"d8058f31-67f2-4f71-8f17-5f839c042a2f\") " Nov 02 18:34:32.497383 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.497281 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8058f31-67f2-4f71-8f17-5f839c042a2f-kube-api-access-nf8km\" (UniqueName: \"kubernetes.io/projected/d8058f31-67f2-4f71-8f17-5f839c042a2f-kube-api-access-nf8km\") pod \"d8058f31-67f2-4f71-8f17-5f839c042a2f\" (UID: \"d8058f31-67f2-4f71-8f17-5f839c042a2f\") " Nov 02 18:34:32.497734 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.497700 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8058f31-67f2-4f71-8f17-5f839c042a2f-openshift-service-ca" pod "d8058f31-67f2-4f71-8f17-5f839c042a2f" (UID: "d8058f31-67f2-4f71-8f17-5f839c042a2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:32.499675 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.499641 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8058f31-67f2-4f71-8f17-5f839c042a2f-metrics-certs" pod "d8058f31-67f2-4f71-8f17-5f839c042a2f" (UID: "d8058f31-67f2-4f71-8f17-5f839c042a2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:34:32.499979 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.499955 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8058f31-67f2-4f71-8f17-5f839c042a2f-maas-api-tls" pod "d8058f31-67f2-4f71-8f17-5f839c042a2f" (UID: "d8058f31-67f2-4f71-8f17-5f839c042a2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:34:32.500107 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.499980 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8058f31-67f2-4f71-8f17-5f839c042a2f-kube-api-access-nf8km" pod "d8058f31-67f2-4f71-8f17-5f839c042a2f" (UID: "d8058f31-67f2-4f71-8f17-5f839c042a2f"). InnerVolumeSpecName "kube-api-access-nf8km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:32.598939 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.598895 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8058f31-67f2-4f71-8f17-5f839c042a2f-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:32.598939 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.598935 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8058f31-67f2-4f71-8f17-5f839c042a2f-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:32.598939 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.598947 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8058f31-67f2-4f71-8f17-5f839c042a2f-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:32.599258 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.598956 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf8km\" (UniqueName: \"kubernetes.io/projected/d8058f31-67f2-4f71-8f17-5f839c042a2f-kube-api-access-nf8km\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:32.792839 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.792804 5601 generic.go:356] "Generic (PLEG): container finished" podID="d8058f31-67f2-4f71-8f17-5f839c042a2f" containerID="cd3ed2b7a2cb6d34a01d609617839afb6a08ef7f4edb83d457ddc6f6c93f734b" exitCode=0 Nov 02 18:34:32.793054 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.792858 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" Nov 02 18:34:32.793054 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.792891 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" event={"ID":"d8058f31-67f2-4f71-8f17-5f839c042a2f","Type":"ContainerDied","Data":"cd3ed2b7a2cb6d34a01d609617839afb6a08ef7f4edb83d457ddc6f6c93f734b"} Nov 02 18:34:32.793054 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.792945 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22" event={"ID":"d8058f31-67f2-4f71-8f17-5f839c042a2f","Type":"ContainerDied","Data":"98b82c902f48315348bc41b2c26046d4e0ac3083a6957d1f722e4ce4a4f461f0"} Nov 02 18:34:32.793054 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.792967 5601 scope.go:122] "RemoveContainer" containerID="cd3ed2b7a2cb6d34a01d609617839afb6a08ef7f4edb83d457ddc6f6c93f734b" Nov 02 18:34:32.803530 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.803505 5601 scope.go:122] "RemoveContainer" containerID="cd3ed2b7a2cb6d34a01d609617839afb6a08ef7f4edb83d457ddc6f6c93f734b" Nov 02 18:34:32.803866 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:34:32.803842 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd3ed2b7a2cb6d34a01d609617839afb6a08ef7f4edb83d457ddc6f6c93f734b\": container with ID starting with cd3ed2b7a2cb6d34a01d609617839afb6a08ef7f4edb83d457ddc6f6c93f734b not found: ID does not exist" containerID="cd3ed2b7a2cb6d34a01d609617839afb6a08ef7f4edb83d457ddc6f6c93f734b" Nov 02 18:34:32.803952 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.803874 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3ed2b7a2cb6d34a01d609617839afb6a08ef7f4edb83d457ddc6f6c93f734b"} err="failed to get container status \"cd3ed2b7a2cb6d34a01d609617839afb6a08ef7f4edb83d457ddc6f6c93f734b\": rpc error: code = NotFound desc = could not find container \"cd3ed2b7a2cb6d34a01d609617839afb6a08ef7f4edb83d457ddc6f6c93f734b\": container with ID starting with cd3ed2b7a2cb6d34a01d609617839afb6a08ef7f4edb83d457ddc6f6c93f734b not found: ID does not exist" Nov 02 18:34:32.817582 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.817550 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22"] Nov 02 18:34:32.819111 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:32.819083 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9165c3-54597f868b-wxd22"] Nov 02 18:34:33.424602 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:33.424535 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8058f31-67f2-4f71-8f17-5f839c042a2f" path="/var/lib/kubelet/pods/d8058f31-67f2-4f71-8f17-5f839c042a2f/volumes" Nov 02 18:34:34.485713 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:34:34.485669 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:34:38.894855 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:38.894807 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr"] Nov 02 18:34:38.895426 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:38.895106 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" podUID="f398ebca-880e-42a8-bb69-9352537dddff" containerName="istio-proxy" containerID="cri-o://9bd937518ff1ddba8163957721784adf506596ef3cd25a33ce60b7c800f1be65" gracePeriod=30 Nov 02 18:34:44.153825 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.153795 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 18:34:44.224491 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.224451 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-istio-data\") pod \"f398ebca-880e-42a8-bb69-9352537dddff\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " Nov 02 18:34:44.224491 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.224496 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-istio-envoy\") pod \"f398ebca-880e-42a8-bb69-9352537dddff\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " Nov 02 18:34:44.224811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.224547 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-workload-socket\") pod \"f398ebca-880e-42a8-bb69-9352537dddff\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " Nov 02 18:34:44.224811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.224567 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-credential-socket\") pod \"f398ebca-880e-42a8-bb69-9352537dddff\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " Nov 02 18:34:44.224811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.224610 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f398ebca-880e-42a8-bb69-9352537dddff-kube-api-access-rpr8f\" (UniqueName: \"kubernetes.io/projected/f398ebca-880e-42a8-bb69-9352537dddff-kube-api-access-rpr8f\") pod \"f398ebca-880e-42a8-bb69-9352537dddff\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " Nov 02 18:34:44.224811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.224669 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f398ebca-880e-42a8-bb69-9352537dddff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f398ebca-880e-42a8-bb69-9352537dddff-istio-podinfo\") pod \"f398ebca-880e-42a8-bb69-9352537dddff\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " Nov 02 18:34:44.224811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.224725 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-workload-certs\") pod \"f398ebca-880e-42a8-bb69-9352537dddff\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " Nov 02 18:34:44.225104 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.224815 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-istio-data" pod "f398ebca-880e-42a8-bb69-9352537dddff" (UID: "f398ebca-880e-42a8-bb69-9352537dddff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:44.225104 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.224851 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f398ebca-880e-42a8-bb69-9352537dddff-istio-token\" (UniqueName: \"kubernetes.io/projected/f398ebca-880e-42a8-bb69-9352537dddff-istio-token\") pod \"f398ebca-880e-42a8-bb69-9352537dddff\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " Nov 02 18:34:44.225104 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.224932 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f398ebca-880e-42a8-bb69-9352537dddff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f398ebca-880e-42a8-bb69-9352537dddff-istiod-ca-cert\") pod \"f398ebca-880e-42a8-bb69-9352537dddff\" (UID: \"f398ebca-880e-42a8-bb69-9352537dddff\") " Nov 02 18:34:44.225104 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.224950 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-workload-socket" pod "f398ebca-880e-42a8-bb69-9352537dddff" (UID: "f398ebca-880e-42a8-bb69-9352537dddff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:44.225104 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.224988 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-credential-socket" pod "f398ebca-880e-42a8-bb69-9352537dddff" (UID: "f398ebca-880e-42a8-bb69-9352537dddff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:44.225104 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.225073 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-workload-certs" pod "f398ebca-880e-42a8-bb69-9352537dddff" (UID: "f398ebca-880e-42a8-bb69-9352537dddff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:44.225405 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.225332 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:44.225405 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.225352 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:44.225405 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.225367 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:44.225405 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.225382 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:44.225405 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.225387 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f398ebca-880e-42a8-bb69-9352537dddff-istiod-ca-cert" pod "f398ebca-880e-42a8-bb69-9352537dddff" (UID: "f398ebca-880e-42a8-bb69-9352537dddff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:44.226906 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.226880 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-istio-envoy" pod "f398ebca-880e-42a8-bb69-9352537dddff" (UID: "f398ebca-880e-42a8-bb69-9352537dddff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:44.227054 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.227029 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f398ebca-880e-42a8-bb69-9352537dddff-istio-token" pod "f398ebca-880e-42a8-bb69-9352537dddff" (UID: "f398ebca-880e-42a8-bb69-9352537dddff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:44.227356 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.227323 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f398ebca-880e-42a8-bb69-9352537dddff-kube-api-access-rpr8f" pod "f398ebca-880e-42a8-bb69-9352537dddff" (UID: "f398ebca-880e-42a8-bb69-9352537dddff"). InnerVolumeSpecName "kube-api-access-rpr8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:44.227356 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.227329 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f398ebca-880e-42a8-bb69-9352537dddff-istio-podinfo" pod "f398ebca-880e-42a8-bb69-9352537dddff" (UID: "f398ebca-880e-42a8-bb69-9352537dddff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:34:44.326221 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.326179 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f398ebca-880e-42a8-bb69-9352537dddff-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:44.326221 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.326214 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpr8f\" (UniqueName: \"kubernetes.io/projected/f398ebca-880e-42a8-bb69-9352537dddff-kube-api-access-rpr8f\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:44.326221 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.326226 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f398ebca-880e-42a8-bb69-9352537dddff-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:44.326485 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.326235 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f398ebca-880e-42a8-bb69-9352537dddff-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:44.326485 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.326244 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f398ebca-880e-42a8-bb69-9352537dddff-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:44.463488 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.463447 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s"] Nov 02 18:34:44.468230 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.468186 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9"] Nov 02 18:34:44.866432 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.866398 5601 generic.go:356] "Generic (PLEG): container finished" podID="f398ebca-880e-42a8-bb69-9352537dddff" containerID="9bd937518ff1ddba8163957721784adf506596ef3cd25a33ce60b7c800f1be65" exitCode=0 Nov 02 18:34:44.866642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.866438 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" event={"ID":"f398ebca-880e-42a8-bb69-9352537dddff","Type":"ContainerDied","Data":"9bd937518ff1ddba8163957721784adf506596ef3cd25a33ce60b7c800f1be65"} Nov 02 18:34:44.866642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.866445 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" Nov 02 18:34:44.866642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.866476 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr" event={"ID":"f398ebca-880e-42a8-bb69-9352537dddff","Type":"ContainerDied","Data":"8228391266fe0b062292a0ba1a27742457edce3e470e947c662892013a266ad6"} Nov 02 18:34:44.866642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.866497 5601 scope.go:122] "RemoveContainer" containerID="9bd937518ff1ddba8163957721784adf506596ef3cd25a33ce60b7c800f1be65" Nov 02 18:34:44.876741 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.876721 5601 scope.go:122] "RemoveContainer" containerID="9bd937518ff1ddba8163957721784adf506596ef3cd25a33ce60b7c800f1be65" Nov 02 18:34:44.877068 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:34:44.877044 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd937518ff1ddba8163957721784adf506596ef3cd25a33ce60b7c800f1be65\": container with ID starting with 9bd937518ff1ddba8163957721784adf506596ef3cd25a33ce60b7c800f1be65 not found: ID does not exist" containerID="9bd937518ff1ddba8163957721784adf506596ef3cd25a33ce60b7c800f1be65" Nov 02 18:34:44.877171 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.877076 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd937518ff1ddba8163957721784adf506596ef3cd25a33ce60b7c800f1be65"} err="failed to get container status \"9bd937518ff1ddba8163957721784adf506596ef3cd25a33ce60b7c800f1be65\": rpc error: code = NotFound desc = could not find container \"9bd937518ff1ddba8163957721784adf506596ef3cd25a33ce60b7c800f1be65\": container with ID starting with 9bd937518ff1ddba8163957721784adf506596ef3cd25a33ce60b7c800f1be65 not found: ID does not exist" Nov 02 18:34:44.889265 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.889238 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr"] Nov 02 18:34:44.893795 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:44.893768 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-9v6sr"] Nov 02 18:34:45.426405 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:45.426361 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f398ebca-880e-42a8-bb69-9352537dddff" path="/var/lib/kubelet/pods/f398ebca-880e-42a8-bb69-9352537dddff/volumes" Nov 02 18:34:56.424602 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.424563 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl"] Nov 02 18:34:56.425115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.424979 5601 cpu_manager.go:414] "containerMap: removing container" podUID="ad96fe9c-98b9-41bb-b532-1b3ca4da572c" containerName="istio-proxy" Nov 02 18:34:56.425115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.424992 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad96fe9c-98b9-41bb-b532-1b3ca4da572c" containerName="istio-proxy" Nov 02 18:34:56.425115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425006 5601 cpu_manager.go:414] "containerMap: removing container" podUID="cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" containerName="istio-proxy" Nov 02 18:34:56.425115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425013 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" containerName="istio-proxy" Nov 02 18:34:56.425115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425027 5601 cpu_manager.go:414] "containerMap: removing container" podUID="f26c04ee-0060-4863-b50f-2eed99aad8b1" containerName="istio-proxy" Nov 02 18:34:56.425115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425033 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f26c04ee-0060-4863-b50f-2eed99aad8b1" containerName="istio-proxy" Nov 02 18:34:56.425115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425039 5601 cpu_manager.go:414] "containerMap: removing container" podUID="13f70b10-e5fb-4844-b55d-5883abbfd8e5" containerName="maas-api" Nov 02 18:34:56.425115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425043 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f70b10-e5fb-4844-b55d-5883abbfd8e5" containerName="maas-api" Nov 02 18:34:56.425115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425053 5601 cpu_manager.go:414] "containerMap: removing container" podUID="f26c04ee-0060-4863-b50f-2eed99aad8b1" containerName="istio-proxy" Nov 02 18:34:56.425115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425061 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f26c04ee-0060-4863-b50f-2eed99aad8b1" containerName="istio-proxy" Nov 02 18:34:56.425115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425068 5601 cpu_manager.go:414] "containerMap: removing container" podUID="ac238df9-b947-47d0-a52a-38145c23f61a" containerName="cleanup" Nov 02 18:34:56.425115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425073 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac238df9-b947-47d0-a52a-38145c23f61a" containerName="cleanup" Nov 02 18:34:56.425115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425079 5601 cpu_manager.go:414] "containerMap: removing container" podUID="c2852736-70fa-4810-9f5a-6211fae2d2c9" containerName="maas-api" Nov 02 18:34:56.425115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425084 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2852736-70fa-4810-9f5a-6211fae2d2c9" containerName="maas-api" Nov 02 18:34:56.425115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425092 5601 cpu_manager.go:414] "containerMap: removing container" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" Nov 02 18:34:56.425115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425100 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" Nov 02 18:34:56.425115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425108 5601 cpu_manager.go:414] "containerMap: removing container" podUID="d8058f31-67f2-4f71-8f17-5f839c042a2f" containerName="maas-api" Nov 02 18:34:56.425115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425112 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8058f31-67f2-4f71-8f17-5f839c042a2f" containerName="maas-api" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425138 5601 cpu_manager.go:414] "containerMap: removing container" podUID="adb37323-1f64-44d2-906c-853342775292" containerName="payload-pre-processing" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425143 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb37323-1f64-44d2-906c-853342775292" containerName="payload-pre-processing" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425150 5601 cpu_manager.go:414] "containerMap: removing container" podUID="4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99" containerName="cleanup" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425155 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a50c6ec-b4b0-4a02-98c2-a2b5f5d7bd99" containerName="cleanup" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425162 5601 cpu_manager.go:414] "containerMap: removing container" podUID="ad96fe9c-98b9-41bb-b532-1b3ca4da572c" containerName="istio-proxy" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425168 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad96fe9c-98b9-41bb-b532-1b3ca4da572c" containerName="istio-proxy" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425181 5601 cpu_manager.go:414] "containerMap: removing container" podUID="f398ebca-880e-42a8-bb69-9352537dddff" containerName="istio-proxy" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425187 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f398ebca-880e-42a8-bb69-9352537dddff" containerName="istio-proxy" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425195 5601 cpu_manager.go:414] "containerMap: removing container" podUID="db8e57a9-62e6-4229-921d-f5a40cde2a7a" containerName="istio-proxy" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425200 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db8e57a9-62e6-4229-921d-f5a40cde2a7a" containerName="istio-proxy" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425208 5601 cpu_manager.go:414] "containerMap: removing container" podUID="48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" containerName="istio-proxy" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425212 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" containerName="istio-proxy" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425223 5601 cpu_manager.go:414] "containerMap: removing container" podUID="e30e6af9-02a5-4b4f-900a-b1de3b3daae9" containerName="istio-proxy" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425227 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e30e6af9-02a5-4b4f-900a-b1de3b3daae9" containerName="istio-proxy" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425235 5601 cpu_manager.go:414] "containerMap: removing container" podUID="d098a18f-93d8-4958-833b-252443a3b8b8" containerName="main" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425239 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d098a18f-93d8-4958-833b-252443a3b8b8" containerName="main" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425253 5601 cpu_manager.go:414] "containerMap: removing container" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425261 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425267 5601 cpu_manager.go:414] "containerMap: removing container" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425271 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425281 5601 cpu_manager.go:414] "containerMap: removing container" podUID="01186ed8-9a15-4802-924a-85dd46e7dc8b" containerName="cleanup" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425286 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01186ed8-9a15-4802-924a-85dd46e7dc8b" containerName="cleanup" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425294 5601 cpu_manager.go:414] "containerMap: removing container" podUID="deb3a82f-14e6-4cdf-8fa7-12db317f98f8" containerName="cleanup" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425299 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb3a82f-14e6-4cdf-8fa7-12db317f98f8" containerName="cleanup" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425361 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="db8e57a9-62e6-4229-921d-f5a40cde2a7a" containerName="istio-proxy" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425370 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad96fe9c-98b9-41bb-b532-1b3ca4da572c" containerName="istio-proxy" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425377 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfef12c4-5397-4b54-8d3a-cb4ad6ec8c7a" containerName="istio-proxy" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425384 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="f26c04ee-0060-4863-b50f-2eed99aad8b1" containerName="istio-proxy" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425389 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f70b10-e5fb-4844-b55d-5883abbfd8e5" containerName="maas-api" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425396 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="f26c04ee-0060-4863-b50f-2eed99aad8b1" containerName="istio-proxy" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425402 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac238df9-b947-47d0-a52a-38145c23f61a" containerName="cleanup" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425413 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2852736-70fa-4810-9f5a-6211fae2d2c9" containerName="maas-api" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425418 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d5a3bb4-a17b-41cf-b512-f4de4bd0e471" containerName="cleanup" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425424 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde15228-bd90-4a8a-a877-cac31762b285" containerName="main" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425428 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8058f31-67f2-4f71-8f17-5f839c042a2f" containerName="maas-api" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425433 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb3a82f-14e6-4cdf-8fa7-12db317f98f8" containerName="cleanup" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425438 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb37323-1f64-44d2-906c-853342775292" containerName="payload-pre-processing" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425443 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="f398ebca-880e-42a8-bb69-9352537dddff" containerName="istio-proxy" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425449 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a8ce0d-ecf7-4f11-a779-c0b60d3dfcc4" containerName="istio-proxy" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425455 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="e30e6af9-02a5-4b4f-900a-b1de3b3daae9" containerName="istio-proxy" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425459 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="d098a18f-93d8-4958-833b-252443a3b8b8" containerName="main" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425466 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="76064488-6a4c-4f85-b1ee-75388968d2a1" containerName="main" Nov 02 18:34:56.425761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.425471 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d6afd6-8aaa-40b6-9728-4ddc5aba3aff" containerName="main" Nov 02 18:34:56.430471 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.430451 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.435043 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.435020 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-77f103-openshift-default-dockercfg-v9fvh\"" Nov 02 18:34:56.445382 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.445360 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl"] Nov 02 18:34:56.456224 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.456194 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbcnz\" (UniqueName: \"kubernetes.io/projected/cbf8a0ca-cff7-4045-93de-28767ebf698c-kube-api-access-qbcnz\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.456384 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.456247 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.456384 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.456278 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.456384 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.456312 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.456384 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.456343 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.456384 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.456370 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.457732 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.456482 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.457732 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.456530 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf8a0ca-cff7-4045-93de-28767ebf698c-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.457732 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.457085 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.557774 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.557732 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf8a0ca-cff7-4045-93de-28767ebf698c-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.557990 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.557813 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.557990 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.557847 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbcnz\" (UniqueName: \"kubernetes.io/projected/cbf8a0ca-cff7-4045-93de-28767ebf698c-kube-api-access-qbcnz\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.557990 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.557874 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.557990 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.557904 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.557990 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.557938 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.557990 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.557972 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.558369 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.557997 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.558369 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.558103 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.558746 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.558536 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.558746 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.558570 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.558746 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.558642 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.558746 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.558666 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf8a0ca-cff7-4045-93de-28767ebf698c-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.559021 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.558835 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.561196 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.561173 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.561390 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.561368 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.565984 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.565952 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.566093 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.566063 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbcnz\" (UniqueName: \"kubernetes.io/projected/cbf8a0ca-cff7-4045-93de-28767ebf698c-kube-api-access-qbcnz\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:56.747178 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:56.747101 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:57.022573 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:57.022545 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl"] Nov 02 18:34:57.025013 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:34:57.024986 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf8a0ca_cff7_4045_93de_28767ebf698c.slice/crio-71e41532322a597bab14455e9e82086498a8b534ba56d0377ebbdfd028e93bcb WatchSource:0}: Error finding container 71e41532322a597bab14455e9e82086498a8b534ba56d0377ebbdfd028e93bcb: Status 404 returned error can't find the container with id 71e41532322a597bab14455e9e82086498a8b534ba56d0377ebbdfd028e93bcb Nov 02 18:34:57.027106 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:57.027070 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:34:57.027216 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:57.027132 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:34:57.027216 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:57.027159 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:34:57.964998 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:57.964953 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" event={"ID":"cbf8a0ca-cff7-4045-93de-28767ebf698c","Type":"ContainerStarted","Data":"f1b8700cf9f5ef3bf53f775ddd8f3349e5c405eae4557e6e1bece2b88fe34fc4"} Nov 02 18:34:57.965399 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:57.965016 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" event={"ID":"cbf8a0ca-cff7-4045-93de-28767ebf698c","Type":"ContainerStarted","Data":"71e41532322a597bab14455e9e82086498a8b534ba56d0377ebbdfd028e93bcb"} Nov 02 18:34:57.987658 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:57.987592 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" podStartSLOduration=1.987578869 podStartE2EDuration="1.987578869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:34:57.98505933 +0000 UTC m=+3662.676113347" watchObservedRunningTime="2026-11-02 18:34:57.987578869 +0000 UTC m=+3662.678632933" Nov 02 18:34:58.752641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:58.749985 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:34:58.758641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:58.757731 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 02 18:34:58.758641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:58.757797 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" podUID="cbf8a0ca-cff7-4045-93de-28767ebf698c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 02 18:34:59.485724 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.485682 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" containerID="cri-o://f29c3a86d5ed3b1914566cc097be0ca9b52d424bd7b408ddd4b8c5172fbf81fb" gracePeriod=45 Nov 02 18:34:59.488824 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.488792 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" containerID="cri-o://b70697e3ce84ae0a370376e8afc1c8c3d96021b15e1f17fdea3cf2d3441927a4" gracePeriod=45 Nov 02 18:34:59.760257 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.751051 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 02 18:34:59.760257 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.751118 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" podUID="cbf8a0ca-cff7-4045-93de-28767ebf698c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 02 18:34:59.826657 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.824336 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 18:34:59.831788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.831762 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 18:34:59.888722 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.888689 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b595c7d-1c54-412a-9035-79633dcd1f24-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b595c7d-1c54-412a-9035-79633dcd1f24-tls-certs\") pod \"4b595c7d-1c54-412a-9035-79633dcd1f24\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " Nov 02 18:34:59.888722 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.888727 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-dshm\") pod \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " Nov 02 18:34:59.888993 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.888793 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-dshm\") pod \"4b595c7d-1c54-412a-9035-79633dcd1f24\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " Nov 02 18:34:59.888993 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.888817 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-model-cache\") pod \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " Nov 02 18:34:59.888993 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.888841 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-home\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-home\") pod \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " Nov 02 18:34:59.888993 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.888868 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b595c7d-1c54-412a-9035-79633dcd1f24-kube-api-access-6zrjf\" (UniqueName: \"kubernetes.io/projected/4b595c7d-1c54-412a-9035-79633dcd1f24-kube-api-access-6zrjf\") pod \"4b595c7d-1c54-412a-9035-79633dcd1f24\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " Nov 02 18:34:59.888993 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.888891 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-tmp-dir\") pod \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " Nov 02 18:34:59.888993 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.888907 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-tmp-dir\") pod \"4b595c7d-1c54-412a-9035-79633dcd1f24\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " Nov 02 18:34:59.888993 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.888935 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-tls-certs\") pod \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " Nov 02 18:34:59.888993 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.888957 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-kube-api-access-7q9tm\" (UniqueName: \"kubernetes.io/projected/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-kube-api-access-7q9tm\") pod \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\" (UID: \"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed\") " Nov 02 18:34:59.889436 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.889356 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-home" pod "bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" (UID: "bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:59.889658 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.888995 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-model-cache\") pod \"4b595c7d-1c54-412a-9035-79633dcd1f24\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " Nov 02 18:34:59.889757 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.889693 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-home\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-home\") pod \"4b595c7d-1c54-412a-9035-79633dcd1f24\" (UID: \"4b595c7d-1c54-412a-9035-79633dcd1f24\") " Nov 02 18:34:59.890154 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.890129 5601 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-home\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:59.890545 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.890266 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-tmp-dir" pod "bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" (UID: "bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:59.890545 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.890336 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-home" pod "4b595c7d-1c54-412a-9035-79633dcd1f24" (UID: "4b595c7d-1c54-412a-9035-79633dcd1f24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:59.890545 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.890433 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-model-cache" pod "bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" (UID: "bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:59.890801 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.890695 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-tmp-dir" pod "4b595c7d-1c54-412a-9035-79633dcd1f24" (UID: "4b595c7d-1c54-412a-9035-79633dcd1f24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:59.890940 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.890882 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-model-cache" pod "4b595c7d-1c54-412a-9035-79633dcd1f24" (UID: "4b595c7d-1c54-412a-9035-79633dcd1f24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:59.892349 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.892316 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-tls-certs" pod "bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" (UID: "bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:34:59.892547 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.892450 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b595c7d-1c54-412a-9035-79633dcd1f24-kube-api-access-6zrjf" pod "4b595c7d-1c54-412a-9035-79633dcd1f24" (UID: "4b595c7d-1c54-412a-9035-79633dcd1f24"). InnerVolumeSpecName "kube-api-access-6zrjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:59.892547 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.892521 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-dshm" pod "4b595c7d-1c54-412a-9035-79633dcd1f24" (UID: "4b595c7d-1c54-412a-9035-79633dcd1f24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:59.892708 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.892677 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-dshm" pod "bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" (UID: "bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:59.893368 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.893344 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b595c7d-1c54-412a-9035-79633dcd1f24-tls-certs" pod "4b595c7d-1c54-412a-9035-79633dcd1f24" (UID: "4b595c7d-1c54-412a-9035-79633dcd1f24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:34:59.893585 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.893564 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-kube-api-access-7q9tm" pod "bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" (UID: "bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed"). InnerVolumeSpecName "kube-api-access-7q9tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:59.977058 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.977020 5601 generic.go:356] "Generic (PLEG): container finished" podID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerID="f29c3a86d5ed3b1914566cc097be0ca9b52d424bd7b408ddd4b8c5172fbf81fb" exitCode=0 Nov 02 18:34:59.977267 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.977066 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" Nov 02 18:34:59.977267 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.977101 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" event={"ID":"4b595c7d-1c54-412a-9035-79633dcd1f24","Type":"ContainerDied","Data":"f29c3a86d5ed3b1914566cc097be0ca9b52d424bd7b408ddd4b8c5172fbf81fb"} Nov 02 18:34:59.977267 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.977155 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s" event={"ID":"4b595c7d-1c54-412a-9035-79633dcd1f24","Type":"ContainerDied","Data":"8b55d4baea151a59fa780426a6da3db91caff5fe65f435b51296747a4bfb3bdd"} Nov 02 18:34:59.977267 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.977179 5601 scope.go:122] "RemoveContainer" containerID="f29c3a86d5ed3b1914566cc097be0ca9b52d424bd7b408ddd4b8c5172fbf81fb" Nov 02 18:34:59.978750 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.978727 5601 generic.go:356] "Generic (PLEG): container finished" podID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerID="b70697e3ce84ae0a370376e8afc1c8c3d96021b15e1f17fdea3cf2d3441927a4" exitCode=0 Nov 02 18:34:59.978896 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.978762 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" Nov 02 18:34:59.978896 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.978811 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" event={"ID":"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed","Type":"ContainerDied","Data":"b70697e3ce84ae0a370376e8afc1c8c3d96021b15e1f17fdea3cf2d3441927a4"} Nov 02 18:34:59.978896 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.978851 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" event={"ID":"bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed","Type":"ContainerDied","Data":"16b8c0b4f78ae234a5bbdd1b67a362b6c4eb3821af85730253f75cfb8ea93e1f"} Nov 02 18:34:59.989937 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.989920 5601 scope.go:122] "RemoveContainer" containerID="f29c3a86d5ed3b1914566cc097be0ca9b52d424bd7b408ddd4b8c5172fbf81fb" Nov 02 18:34:59.990217 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:34:59.990197 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f29c3a86d5ed3b1914566cc097be0ca9b52d424bd7b408ddd4b8c5172fbf81fb\": container with ID starting with f29c3a86d5ed3b1914566cc097be0ca9b52d424bd7b408ddd4b8c5172fbf81fb not found: ID does not exist" containerID="f29c3a86d5ed3b1914566cc097be0ca9b52d424bd7b408ddd4b8c5172fbf81fb" Nov 02 18:34:59.990378 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.990223 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f29c3a86d5ed3b1914566cc097be0ca9b52d424bd7b408ddd4b8c5172fbf81fb"} err="failed to get container status \"f29c3a86d5ed3b1914566cc097be0ca9b52d424bd7b408ddd4b8c5172fbf81fb\": rpc error: code = NotFound desc = could not find container \"f29c3a86d5ed3b1914566cc097be0ca9b52d424bd7b408ddd4b8c5172fbf81fb\": container with ID starting with f29c3a86d5ed3b1914566cc097be0ca9b52d424bd7b408ddd4b8c5172fbf81fb not found: ID does not exist" Nov 02 18:34:59.990458 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.990446 5601 scope.go:122] "RemoveContainer" containerID="b70697e3ce84ae0a370376e8afc1c8c3d96021b15e1f17fdea3cf2d3441927a4" Nov 02 18:34:59.990607 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.990590 5601 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-dshm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:59.990692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.990610 5601 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-model-cache\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:59.990692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.990635 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zrjf\" (UniqueName: \"kubernetes.io/projected/4b595c7d-1c54-412a-9035-79633dcd1f24-kube-api-access-6zrjf\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:59.990692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.990646 5601 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-tmp-dir\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:59.990692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.990659 5601 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-tmp-dir\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:59.990692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.990668 5601 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-tls-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:59.990692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.990677 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q9tm\" (UniqueName: \"kubernetes.io/projected/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-kube-api-access-7q9tm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:59.990692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.990686 5601 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-model-cache\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:59.990692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.990694 5601 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b595c7d-1c54-412a-9035-79633dcd1f24-home\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:59.991150 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.990706 5601 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b595c7d-1c54-412a-9035-79633dcd1f24-tls-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:59.991150 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.990721 5601 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed-dshm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:34:59.998687 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.998665 5601 scope.go:122] "RemoveContainer" containerID="b70697e3ce84ae0a370376e8afc1c8c3d96021b15e1f17fdea3cf2d3441927a4" Nov 02 18:34:59.998952 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:34:59.998926 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70697e3ce84ae0a370376e8afc1c8c3d96021b15e1f17fdea3cf2d3441927a4\": container with ID starting with b70697e3ce84ae0a370376e8afc1c8c3d96021b15e1f17fdea3cf2d3441927a4 not found: ID does not exist" containerID="b70697e3ce84ae0a370376e8afc1c8c3d96021b15e1f17fdea3cf2d3441927a4" Nov 02 18:34:59.999015 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:34:59.998959 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70697e3ce84ae0a370376e8afc1c8c3d96021b15e1f17fdea3cf2d3441927a4"} err="failed to get container status \"b70697e3ce84ae0a370376e8afc1c8c3d96021b15e1f17fdea3cf2d3441927a4\": rpc error: code = NotFound desc = could not find container \"b70697e3ce84ae0a370376e8afc1c8c3d96021b15e1f17fdea3cf2d3441927a4\": container with ID starting with b70697e3ce84ae0a370376e8afc1c8c3d96021b15e1f17fdea3cf2d3441927a4 not found: ID does not exist" Nov 02 18:35:00.008558 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:00.005157 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s"] Nov 02 18:35:00.013444 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:00.012650 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/distinct-2-w4-77f103-kserve-68f97f49fb-6478s"] Nov 02 18:35:00.022404 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:00.022378 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9"] Nov 02 18:35:00.025756 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:00.025735 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9"] Nov 02 18:35:00.753736 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:00.751931 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 02 18:35:00.753736 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:00.752013 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" podUID="cbf8a0ca-cff7-4045-93de-28767ebf698c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 02 18:35:00.821863 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:00.821808 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/distinct-w4-77f103-kserve-f865fd7f9-fhjj9" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:35:01.424424 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:01.424386 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" path="/var/lib/kubelet/pods/4b595c7d-1c54-412a-9035-79633dcd1f24/volumes" Nov 02 18:35:01.424848 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:01.424834 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" path="/var/lib/kubelet/pods/bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed/volumes" Nov 02 18:35:01.756253 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:01.756145 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 02 18:35:01.756892 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:01.756837 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" podUID="cbf8a0ca-cff7-4045-93de-28767ebf698c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 02 18:35:01.889949 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:01.889906 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt"] Nov 02 18:35:01.890201 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:01.890169 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" podUID="3416b5e2-7a44-4815-8420-cb6ed9f2e3d7" containerName="maas-api" containerID="cri-o://f7ede08cda3f6cf87edc881129ff8f2e5c5e2a2ca0ae676ebfd7b4ff4bc0d862" gracePeriod=30 Nov 02 18:35:02.206221 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:02.206192 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" Nov 02 18:35:02.223672 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:02.223584 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-kube-api-access-s9mn8\" (UniqueName: \"kubernetes.io/projected/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-kube-api-access-s9mn8\") pod \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\" (UID: \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\") " Nov 02 18:35:02.224138 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:02.223692 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-maas-api-tls\") pod \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\" (UID: \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\") " Nov 02 18:35:02.224138 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:02.223741 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-metrics-certs\") pod \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\" (UID: \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\") " Nov 02 18:35:02.224138 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:02.223771 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-openshift-service-ca\") pod \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\" (UID: \"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7\") " Nov 02 18:35:02.224756 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:02.224730 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-openshift-service-ca" pod "3416b5e2-7a44-4815-8420-cb6ed9f2e3d7" (UID: "3416b5e2-7a44-4815-8420-cb6ed9f2e3d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:35:02.227769 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:02.227601 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-metrics-certs" pod "3416b5e2-7a44-4815-8420-cb6ed9f2e3d7" (UID: "3416b5e2-7a44-4815-8420-cb6ed9f2e3d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:35:02.228349 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:02.228315 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-kube-api-access-s9mn8" pod "3416b5e2-7a44-4815-8420-cb6ed9f2e3d7" (UID: "3416b5e2-7a44-4815-8420-cb6ed9f2e3d7"). InnerVolumeSpecName "kube-api-access-s9mn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:35:02.228440 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:02.228367 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-maas-api-tls" pod "3416b5e2-7a44-4815-8420-cb6ed9f2e3d7" (UID: "3416b5e2-7a44-4815-8420-cb6ed9f2e3d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:35:02.325153 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:02.325043 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:02.325153 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:02.325086 5601 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-openshift-service-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:02.325153 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:02.325104 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9mn8\" (UniqueName: \"kubernetes.io/projected/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-kube-api-access-s9mn8\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:02.325153 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:02.325117 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7-maas-api-tls\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:02.754637 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:02.754108 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 02 18:35:02.754637 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:02.754180 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" podUID="cbf8a0ca-cff7-4045-93de-28767ebf698c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 02 18:35:03.000080 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:03.000043 5601 generic.go:356] "Generic (PLEG): container finished" podID="3416b5e2-7a44-4815-8420-cb6ed9f2e3d7" containerID="f7ede08cda3f6cf87edc881129ff8f2e5c5e2a2ca0ae676ebfd7b4ff4bc0d862" exitCode=0 Nov 02 18:35:03.000520 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:03.000109 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" Nov 02 18:35:03.000520 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:03.000124 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" event={"ID":"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7","Type":"ContainerDied","Data":"f7ede08cda3f6cf87edc881129ff8f2e5c5e2a2ca0ae676ebfd7b4ff4bc0d862"} Nov 02 18:35:03.000520 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:03.000182 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt" event={"ID":"3416b5e2-7a44-4815-8420-cb6ed9f2e3d7","Type":"ContainerDied","Data":"ec6b89ef98a17dbc2ef6b5147f49362d728d7e7ea08bc832c9187630d21f8193"} Nov 02 18:35:03.000520 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:03.000204 5601 scope.go:122] "RemoveContainer" containerID="f7ede08cda3f6cf87edc881129ff8f2e5c5e2a2ca0ae676ebfd7b4ff4bc0d862" Nov 02 18:35:03.009823 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:03.009800 5601 scope.go:122] "RemoveContainer" containerID="f7ede08cda3f6cf87edc881129ff8f2e5c5e2a2ca0ae676ebfd7b4ff4bc0d862" Nov 02 18:35:03.010135 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:35:03.010112 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ede08cda3f6cf87edc881129ff8f2e5c5e2a2ca0ae676ebfd7b4ff4bc0d862\": container with ID starting with f7ede08cda3f6cf87edc881129ff8f2e5c5e2a2ca0ae676ebfd7b4ff4bc0d862 not found: ID does not exist" containerID="f7ede08cda3f6cf87edc881129ff8f2e5c5e2a2ca0ae676ebfd7b4ff4bc0d862" Nov 02 18:35:03.010203 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:03.010144 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ede08cda3f6cf87edc881129ff8f2e5c5e2a2ca0ae676ebfd7b4ff4bc0d862"} err="failed to get container status \"f7ede08cda3f6cf87edc881129ff8f2e5c5e2a2ca0ae676ebfd7b4ff4bc0d862\": rpc error: code = NotFound desc = could not find container \"f7ede08cda3f6cf87edc881129ff8f2e5c5e2a2ca0ae676ebfd7b4ff4bc0d862\": container with ID starting with f7ede08cda3f6cf87edc881129ff8f2e5c5e2a2ca0ae676ebfd7b4ff4bc0d862 not found: ID does not exist" Nov 02 18:35:03.026680 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:03.026645 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt"] Nov 02 18:35:03.032295 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:03.032264 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-77f103-65d5d5c77-cdcrt"] Nov 02 18:35:03.440560 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:03.440520 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3416b5e2-7a44-4815-8420-cb6ed9f2e3d7" path="/var/lib/kubelet/pods/3416b5e2-7a44-4815-8420-cb6ed9f2e3d7/volumes" Nov 02 18:35:04.485037 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:35:04.485001 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:35:04.788900 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:04.788798 5601 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:35:04.789070 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:04.788890 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" podUID="cbf8a0ca-cff7-4045-93de-28767ebf698c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:35:05.651422 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:05.651388 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:35:05.651941 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:05.651834 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:35:05.652827 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:05.652802 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:35:24.112355 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:24.112309 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl"] Nov 02 18:35:24.112937 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:24.112567 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" podUID="cbf8a0ca-cff7-4045-93de-28767ebf698c" containerName="istio-proxy" containerID="cri-o://f1b8700cf9f5ef3bf53f775ddd8f3349e5c405eae4557e6e1bece2b88fe34fc4" gracePeriod=30 Nov 02 18:35:26.420224 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:26.420180 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk"] Nov 02 18:35:26.420681 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:26.420392 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" podUID="2f57ca2f-9916-45aa-9523-65e8fd925f15" containerName="istio-proxy" containerID="cri-o://616e48855ee28f64839bb98fd48d9e39dd94c7d13b340d72d80b615200b193c8" gracePeriod=30 Nov 02 18:35:27.378846 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:27.378815 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-t5dg5_aad4749c-18d8-4003-8576-211672ff42e5/manager/0.log" Nov 02 18:35:27.751855 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:27.751817 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-hsptq_3aa1eac9-b956-4608-8c5a-343b37a69d70/manager/0.log" Nov 02 18:35:28.522208 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:28.522179 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56bc685b8f-572rh_2ff5f84a-63fe-4f69-bb19-ede4145e82f1/manager/0.log" Nov 02 18:35:29.363634 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.363593 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:35:29.488117 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.488081 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-podinfo\") pod \"cbf8a0ca-cff7-4045-93de-28767ebf698c\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " Nov 02 18:35:29.488310 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.488127 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-data\") pod \"cbf8a0ca-cff7-4045-93de-28767ebf698c\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " Nov 02 18:35:29.488310 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.488187 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf8a0ca-cff7-4045-93de-28767ebf698c-kube-api-access-qbcnz\" (UniqueName: \"kubernetes.io/projected/cbf8a0ca-cff7-4045-93de-28767ebf698c-kube-api-access-qbcnz\") pod \"cbf8a0ca-cff7-4045-93de-28767ebf698c\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " Nov 02 18:35:29.488310 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.488242 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-workload-certs\") pod \"cbf8a0ca-cff7-4045-93de-28767ebf698c\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " Nov 02 18:35:29.488310 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.488262 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbf8a0ca-cff7-4045-93de-28767ebf698c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf8a0ca-cff7-4045-93de-28767ebf698c-istiod-ca-cert\") pod \"cbf8a0ca-cff7-4045-93de-28767ebf698c\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " Nov 02 18:35:29.488310 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.488303 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-envoy\") pod \"cbf8a0ca-cff7-4045-93de-28767ebf698c\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " Nov 02 18:35:29.488588 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.488342 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-token\" (UniqueName: \"kubernetes.io/projected/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-token\") pod \"cbf8a0ca-cff7-4045-93de-28767ebf698c\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " Nov 02 18:35:29.488588 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.488386 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-credential-socket\") pod \"cbf8a0ca-cff7-4045-93de-28767ebf698c\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " Nov 02 18:35:29.488588 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.488430 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-workload-socket\") pod \"cbf8a0ca-cff7-4045-93de-28767ebf698c\" (UID: \"cbf8a0ca-cff7-4045-93de-28767ebf698c\") " Nov 02 18:35:29.488588 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.488512 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-data" pod "cbf8a0ca-cff7-4045-93de-28767ebf698c" (UID: "cbf8a0ca-cff7-4045-93de-28767ebf698c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:29.488588 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.488524 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-workload-certs" pod "cbf8a0ca-cff7-4045-93de-28767ebf698c" (UID: "cbf8a0ca-cff7-4045-93de-28767ebf698c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:29.488933 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.488767 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-workload-socket" pod "cbf8a0ca-cff7-4045-93de-28767ebf698c" (UID: "cbf8a0ca-cff7-4045-93de-28767ebf698c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:29.488933 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.488788 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf8a0ca-cff7-4045-93de-28767ebf698c-istiod-ca-cert" pod "cbf8a0ca-cff7-4045-93de-28767ebf698c" (UID: "cbf8a0ca-cff7-4045-93de-28767ebf698c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:35:29.488933 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.488833 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:29.488933 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.488851 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:29.489138 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.489011 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-credential-socket" pod "cbf8a0ca-cff7-4045-93de-28767ebf698c" (UID: "cbf8a0ca-cff7-4045-93de-28767ebf698c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:29.490607 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.490579 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-envoy" pod "cbf8a0ca-cff7-4045-93de-28767ebf698c" (UID: "cbf8a0ca-cff7-4045-93de-28767ebf698c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:29.490735 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.490656 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-token" pod "cbf8a0ca-cff7-4045-93de-28767ebf698c" (UID: "cbf8a0ca-cff7-4045-93de-28767ebf698c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:35:29.490878 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.490858 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-podinfo" pod "cbf8a0ca-cff7-4045-93de-28767ebf698c" (UID: "cbf8a0ca-cff7-4045-93de-28767ebf698c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:35:29.490878 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.490873 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf8a0ca-cff7-4045-93de-28767ebf698c-kube-api-access-qbcnz" pod "cbf8a0ca-cff7-4045-93de-28767ebf698c" (UID: "cbf8a0ca-cff7-4045-93de-28767ebf698c"). InnerVolumeSpecName "kube-api-access-qbcnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:35:29.589734 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.589688 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbcnz\" (UniqueName: \"kubernetes.io/projected/cbf8a0ca-cff7-4045-93de-28767ebf698c-kube-api-access-qbcnz\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:29.589734 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.589726 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf8a0ca-cff7-4045-93de-28767ebf698c-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:29.589734 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.589736 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:29.589734 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.589745 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:29.590023 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.589756 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:29.590023 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.589764 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf8a0ca-cff7-4045-93de-28767ebf698c-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:29.590023 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:29.589772 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf8a0ca-cff7-4045-93de-28767ebf698c-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:30.165884 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:30.165846 5601 generic.go:356] "Generic (PLEG): container finished" podID="cbf8a0ca-cff7-4045-93de-28767ebf698c" containerID="f1b8700cf9f5ef3bf53f775ddd8f3349e5c405eae4557e6e1bece2b88fe34fc4" exitCode=0 Nov 02 18:35:30.166061 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:30.165901 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" Nov 02 18:35:30.166061 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:30.165937 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" event={"ID":"cbf8a0ca-cff7-4045-93de-28767ebf698c","Type":"ContainerDied","Data":"f1b8700cf9f5ef3bf53f775ddd8f3349e5c405eae4557e6e1bece2b88fe34fc4"} Nov 02 18:35:30.166061 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:30.165990 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl" event={"ID":"cbf8a0ca-cff7-4045-93de-28767ebf698c","Type":"ContainerDied","Data":"71e41532322a597bab14455e9e82086498a8b534ba56d0377ebbdfd028e93bcb"} Nov 02 18:35:30.166061 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:30.166014 5601 scope.go:122] "RemoveContainer" containerID="f1b8700cf9f5ef3bf53f775ddd8f3349e5c405eae4557e6e1bece2b88fe34fc4" Nov 02 18:35:30.175939 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:30.175916 5601 scope.go:122] "RemoveContainer" containerID="f1b8700cf9f5ef3bf53f775ddd8f3349e5c405eae4557e6e1bece2b88fe34fc4" Nov 02 18:35:30.176248 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:35:30.176223 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b8700cf9f5ef3bf53f775ddd8f3349e5c405eae4557e6e1bece2b88fe34fc4\": container with ID starting with f1b8700cf9f5ef3bf53f775ddd8f3349e5c405eae4557e6e1bece2b88fe34fc4 not found: ID does not exist" containerID="f1b8700cf9f5ef3bf53f775ddd8f3349e5c405eae4557e6e1bece2b88fe34fc4" Nov 02 18:35:30.176359 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:30.176258 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b8700cf9f5ef3bf53f775ddd8f3349e5c405eae4557e6e1bece2b88fe34fc4"} err="failed to get container status \"f1b8700cf9f5ef3bf53f775ddd8f3349e5c405eae4557e6e1bece2b88fe34fc4\": rpc error: code = NotFound desc = could not find container \"f1b8700cf9f5ef3bf53f775ddd8f3349e5c405eae4557e6e1bece2b88fe34fc4\": container with ID starting with f1b8700cf9f5ef3bf53f775ddd8f3349e5c405eae4557e6e1bece2b88fe34fc4 not found: ID does not exist" Nov 02 18:35:30.187243 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:30.187215 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl"] Nov 02 18:35:30.192806 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:30.192777 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-mzlwl"] Nov 02 18:35:31.425347 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.425313 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbf8a0ca-cff7-4045-93de-28767ebf698c" path="/var/lib/kubelet/pods/cbf8a0ca-cff7-4045-93de-28767ebf698c/volumes" Nov 02 18:35:31.672389 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.672361 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 18:35:31.813313 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.813227 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f57ca2f-9916-45aa-9523-65e8fd925f15-kube-api-access-lqdlm\" (UniqueName: \"kubernetes.io/projected/2f57ca2f-9916-45aa-9523-65e8fd925f15-kube-api-access-lqdlm\") pod \"2f57ca2f-9916-45aa-9523-65e8fd925f15\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " Nov 02 18:35:31.813313 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.813276 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-envoy\") pod \"2f57ca2f-9916-45aa-9523-65e8fd925f15\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " Nov 02 18:35:31.813313 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.813315 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-podinfo\") pod \"2f57ca2f-9916-45aa-9523-65e8fd925f15\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " Nov 02 18:35:31.813992 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.813350 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-data\") pod \"2f57ca2f-9916-45aa-9523-65e8fd925f15\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " Nov 02 18:35:31.813992 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.813372 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-workload-socket\") pod \"2f57ca2f-9916-45aa-9523-65e8fd925f15\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " Nov 02 18:35:31.813992 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.813434 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-credential-socket\") pod \"2f57ca2f-9916-45aa-9523-65e8fd925f15\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " Nov 02 18:35:31.813992 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.813507 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-token\" (UniqueName: \"kubernetes.io/projected/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-token\") pod \"2f57ca2f-9916-45aa-9523-65e8fd925f15\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " Nov 02 18:35:31.813992 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.813534 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f57ca2f-9916-45aa-9523-65e8fd925f15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f57ca2f-9916-45aa-9523-65e8fd925f15-istiod-ca-cert\") pod \"2f57ca2f-9916-45aa-9523-65e8fd925f15\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " Nov 02 18:35:31.813992 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.813564 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-workload-certs\") pod \"2f57ca2f-9916-45aa-9523-65e8fd925f15\" (UID: \"2f57ca2f-9916-45aa-9523-65e8fd925f15\") " Nov 02 18:35:31.813992 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.813679 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-data" pod "2f57ca2f-9916-45aa-9523-65e8fd925f15" (UID: "2f57ca2f-9916-45aa-9523-65e8fd925f15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:31.813992 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.813726 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-workload-socket" pod "2f57ca2f-9916-45aa-9523-65e8fd925f15" (UID: "2f57ca2f-9916-45aa-9523-65e8fd925f15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:31.813992 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.813774 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-credential-socket" pod "2f57ca2f-9916-45aa-9523-65e8fd925f15" (UID: "2f57ca2f-9916-45aa-9523-65e8fd925f15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:31.813992 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.813933 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-workload-certs" pod "2f57ca2f-9916-45aa-9523-65e8fd925f15" (UID: "2f57ca2f-9916-45aa-9523-65e8fd925f15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:31.814656 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.813994 5601 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-credential-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:31.814656 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.814018 5601 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-data\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:31.814656 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.814020 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f57ca2f-9916-45aa-9523-65e8fd925f15-istiod-ca-cert" pod "2f57ca2f-9916-45aa-9523-65e8fd925f15" (UID: "2f57ca2f-9916-45aa-9523-65e8fd925f15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:35:31.814656 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.814033 5601 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-workload-socket\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:31.815690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.815653 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-token" pod "2f57ca2f-9916-45aa-9523-65e8fd925f15" (UID: "2f57ca2f-9916-45aa-9523-65e8fd925f15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:35:31.815813 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.815719 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-podinfo" pod "2f57ca2f-9916-45aa-9523-65e8fd925f15" (UID: "2f57ca2f-9916-45aa-9523-65e8fd925f15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:35:31.815967 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.815950 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f57ca2f-9916-45aa-9523-65e8fd925f15-kube-api-access-lqdlm" pod "2f57ca2f-9916-45aa-9523-65e8fd925f15" (UID: "2f57ca2f-9916-45aa-9523-65e8fd925f15"). InnerVolumeSpecName "kube-api-access-lqdlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:35:31.816011 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.815973 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-envoy" pod "2f57ca2f-9916-45aa-9523-65e8fd925f15" (UID: "2f57ca2f-9916-45aa-9523-65e8fd925f15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:31.914628 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.914587 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqdlm\" (UniqueName: \"kubernetes.io/projected/2f57ca2f-9916-45aa-9523-65e8fd925f15-kube-api-access-lqdlm\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:31.914811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.914645 5601 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-envoy\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:31.914811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.914656 5601 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-podinfo\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:31.914811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.914665 5601 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f57ca2f-9916-45aa-9523-65e8fd925f15-istio-token\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:31.914811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.914674 5601 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f57ca2f-9916-45aa-9523-65e8fd925f15-istiod-ca-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:31.914811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:31.914682 5601 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f57ca2f-9916-45aa-9523-65e8fd925f15-workload-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:35:32.185175 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:32.185129 5601 generic.go:356] "Generic (PLEG): container finished" podID="2f57ca2f-9916-45aa-9523-65e8fd925f15" containerID="616e48855ee28f64839bb98fd48d9e39dd94c7d13b340d72d80b615200b193c8" exitCode=0 Nov 02 18:35:32.185358 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:32.185182 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" Nov 02 18:35:32.185358 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:32.185216 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" event={"ID":"2f57ca2f-9916-45aa-9523-65e8fd925f15","Type":"ContainerDied","Data":"616e48855ee28f64839bb98fd48d9e39dd94c7d13b340d72d80b615200b193c8"} Nov 02 18:35:32.185358 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:32.185276 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk" event={"ID":"2f57ca2f-9916-45aa-9523-65e8fd925f15","Type":"ContainerDied","Data":"f980fad24c157423ce6af8646a3b5eaebe85e76f9a0dd787111ca5a420d6f7f6"} Nov 02 18:35:32.185358 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:32.185300 5601 scope.go:122] "RemoveContainer" containerID="616e48855ee28f64839bb98fd48d9e39dd94c7d13b340d72d80b615200b193c8" Nov 02 18:35:32.194919 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:32.194897 5601 scope.go:122] "RemoveContainer" containerID="616e48855ee28f64839bb98fd48d9e39dd94c7d13b340d72d80b615200b193c8" Nov 02 18:35:32.195217 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:35:32.195196 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616e48855ee28f64839bb98fd48d9e39dd94c7d13b340d72d80b615200b193c8\": container with ID starting with 616e48855ee28f64839bb98fd48d9e39dd94c7d13b340d72d80b615200b193c8 not found: ID does not exist" containerID="616e48855ee28f64839bb98fd48d9e39dd94c7d13b340d72d80b615200b193c8" Nov 02 18:35:32.195285 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:32.195231 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616e48855ee28f64839bb98fd48d9e39dd94c7d13b340d72d80b615200b193c8"} err="failed to get container status \"616e48855ee28f64839bb98fd48d9e39dd94c7d13b340d72d80b615200b193c8\": rpc error: code = NotFound desc = could not find container \"616e48855ee28f64839bb98fd48d9e39dd94c7d13b340d72d80b615200b193c8\": container with ID starting with 616e48855ee28f64839bb98fd48d9e39dd94c7d13b340d72d80b615200b193c8 not found: ID does not exist" Nov 02 18:35:32.207442 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:32.207411 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk"] Nov 02 18:35:32.211945 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:32.211914 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mdxxk"] Nov 02 18:35:33.424691 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:35:33.424660 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f57ca2f-9916-45aa-9523-65e8fd925f15" path="/var/lib/kubelet/pods/2f57ca2f-9916-45aa-9523-65e8fd925f15/volumes" Nov 02 18:35:34.483804 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:35:34.483765 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:36:04.484559 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:36:04.484519 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:36:09.933376 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:09.933331 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56bc685b8f-572rh"] Nov 02 18:36:09.934276 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:09.933641 5601 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-56bc685b8f-572rh" podUID="2ff5f84a-63fe-4f69-bb19-ede4145e82f1" containerName="manager" containerID="cri-o://44c7c81b02bb5e835b96ba4a88ce0c94d27ad41accbf394a3826891478d072a1" gracePeriod=30 Nov 02 18:36:10.184413 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:10.184358 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bc685b8f-572rh" Nov 02 18:36:10.288504 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:10.288473 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-webhook-cert\") pod \"2ff5f84a-63fe-4f69-bb19-ede4145e82f1\" (UID: \"2ff5f84a-63fe-4f69-bb19-ede4145e82f1\") " Nov 02 18:36:10.288724 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:10.288571 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-kube-api-access-qkzkg\" (UniqueName: \"kubernetes.io/projected/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-kube-api-access-qkzkg\") pod \"2ff5f84a-63fe-4f69-bb19-ede4145e82f1\" (UID: \"2ff5f84a-63fe-4f69-bb19-ede4145e82f1\") " Nov 02 18:36:10.288724 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:10.288641 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-metrics-certs\") pod \"2ff5f84a-63fe-4f69-bb19-ede4145e82f1\" (UID: \"2ff5f84a-63fe-4f69-bb19-ede4145e82f1\") " Nov 02 18:36:10.290700 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:10.290667 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-metrics-certs" pod "2ff5f84a-63fe-4f69-bb19-ede4145e82f1" (UID: "2ff5f84a-63fe-4f69-bb19-ede4145e82f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:36:10.290814 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:10.290710 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-webhook-cert" pod "2ff5f84a-63fe-4f69-bb19-ede4145e82f1" (UID: "2ff5f84a-63fe-4f69-bb19-ede4145e82f1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:36:10.290814 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:10.290731 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-kube-api-access-qkzkg" pod "2ff5f84a-63fe-4f69-bb19-ede4145e82f1" (UID: "2ff5f84a-63fe-4f69-bb19-ede4145e82f1"). InnerVolumeSpecName "kube-api-access-qkzkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:36:10.389509 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:10.389469 5601 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-webhook-cert\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:10.389509 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:10.389506 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkzkg\" (UniqueName: \"kubernetes.io/projected/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-kube-api-access-qkzkg\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:10.389509 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:10.389517 5601 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff5f84a-63fe-4f69-bb19-ede4145e82f1-metrics-certs\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:36:10.390201 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:10.390173 5601 generic.go:356] "Generic (PLEG): container finished" podID="2ff5f84a-63fe-4f69-bb19-ede4145e82f1" containerID="44c7c81b02bb5e835b96ba4a88ce0c94d27ad41accbf394a3826891478d072a1" exitCode=0 Nov 02 18:36:10.390331 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:10.390219 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56bc685b8f-572rh" Nov 02 18:36:10.390331 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:10.390238 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bc685b8f-572rh" event={"ID":"2ff5f84a-63fe-4f69-bb19-ede4145e82f1","Type":"ContainerDied","Data":"44c7c81b02bb5e835b96ba4a88ce0c94d27ad41accbf394a3826891478d072a1"} Nov 02 18:36:10.390331 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:10.390282 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56bc685b8f-572rh" event={"ID":"2ff5f84a-63fe-4f69-bb19-ede4145e82f1","Type":"ContainerDied","Data":"8f9c3a42f5d411a6ed140a87a60854218c34f8f2ff709307c1a4eb2c354391d0"} Nov 02 18:36:10.390331 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:10.390302 5601 scope.go:122] "RemoveContainer" containerID="44c7c81b02bb5e835b96ba4a88ce0c94d27ad41accbf394a3826891478d072a1" Nov 02 18:36:10.401471 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:10.401451 5601 scope.go:122] "RemoveContainer" containerID="44c7c81b02bb5e835b96ba4a88ce0c94d27ad41accbf394a3826891478d072a1" Nov 02 18:36:10.401796 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:36:10.401775 5601 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c7c81b02bb5e835b96ba4a88ce0c94d27ad41accbf394a3826891478d072a1\": container with ID starting with 44c7c81b02bb5e835b96ba4a88ce0c94d27ad41accbf394a3826891478d072a1 not found: ID does not exist" containerID="44c7c81b02bb5e835b96ba4a88ce0c94d27ad41accbf394a3826891478d072a1" Nov 02 18:36:10.401851 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:10.401805 5601 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c7c81b02bb5e835b96ba4a88ce0c94d27ad41accbf394a3826891478d072a1"} err="failed to get container status \"44c7c81b02bb5e835b96ba4a88ce0c94d27ad41accbf394a3826891478d072a1\": rpc error: code = NotFound desc = could not find container \"44c7c81b02bb5e835b96ba4a88ce0c94d27ad41accbf394a3826891478d072a1\": container with ID starting with 44c7c81b02bb5e835b96ba4a88ce0c94d27ad41accbf394a3826891478d072a1 not found: ID does not exist" Nov 02 18:36:10.413967 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:10.413942 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56bc685b8f-572rh"] Nov 02 18:36:10.417273 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:10.417250 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-56bc685b8f-572rh"] Nov 02 18:36:11.425383 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:11.425341 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff5f84a-63fe-4f69-bb19-ede4145e82f1" path="/var/lib/kubelet/pods/2ff5f84a-63fe-4f69-bb19-ede4145e82f1/volumes" Nov 02 18:36:26.541519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.541478 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt"] Nov 02 18:36:26.542092 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542071 5601 cpu_manager.go:414] "containerMap: removing container" podUID="2ff5f84a-63fe-4f69-bb19-ede4145e82f1" containerName="manager" Nov 02 18:36:26.542151 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542096 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff5f84a-63fe-4f69-bb19-ede4145e82f1" containerName="manager" Nov 02 18:36:26.542151 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542113 5601 cpu_manager.go:414] "containerMap: removing container" podUID="3416b5e2-7a44-4815-8420-cb6ed9f2e3d7" containerName="maas-api" Nov 02 18:36:26.542151 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542120 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3416b5e2-7a44-4815-8420-cb6ed9f2e3d7" containerName="maas-api" Nov 02 18:36:26.542151 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542130 5601 cpu_manager.go:414] "containerMap: removing container" podUID="db8e57a9-62e6-4229-921d-f5a40cde2a7a" containerName="istio-proxy" Nov 02 18:36:26.542151 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542138 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db8e57a9-62e6-4229-921d-f5a40cde2a7a" containerName="istio-proxy" Nov 02 18:36:26.542415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542182 5601 cpu_manager.go:414] "containerMap: removing container" podUID="2f57ca2f-9916-45aa-9523-65e8fd925f15" containerName="istio-proxy" Nov 02 18:36:26.542415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542190 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f57ca2f-9916-45aa-9523-65e8fd925f15" containerName="istio-proxy" Nov 02 18:36:26.542415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542204 5601 cpu_manager.go:414] "containerMap: removing container" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" Nov 02 18:36:26.542415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542212 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" Nov 02 18:36:26.542415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542223 5601 cpu_manager.go:414] "containerMap: removing container" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" Nov 02 18:36:26.542415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542231 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" Nov 02 18:36:26.542415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542245 5601 cpu_manager.go:414] "containerMap: removing container" podUID="cbf8a0ca-cff7-4045-93de-28767ebf698c" containerName="istio-proxy" Nov 02 18:36:26.542415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542251 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbf8a0ca-cff7-4045-93de-28767ebf698c" containerName="istio-proxy" Nov 02 18:36:26.542415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542349 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="db8e57a9-62e6-4229-921d-f5a40cde2a7a" containerName="istio-proxy" Nov 02 18:36:26.542415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542360 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f57ca2f-9916-45aa-9523-65e8fd925f15" containerName="istio-proxy" Nov 02 18:36:26.542415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542368 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad96fe9c-98b9-41bb-b532-1b3ca4da572c" containerName="istio-proxy" Nov 02 18:36:26.542415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542377 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b595c7d-1c54-412a-9035-79633dcd1f24" containerName="main" Nov 02 18:36:26.542415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542387 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdb98d3f-fb16-46dd-8c9d-8932a8fc47ed" containerName="main" Nov 02 18:36:26.542415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542396 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbf8a0ca-cff7-4045-93de-28767ebf698c" containerName="istio-proxy" Nov 02 18:36:26.542415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542408 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff5f84a-63fe-4f69-bb19-ede4145e82f1" containerName="manager" Nov 02 18:36:26.542415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.542416 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="3416b5e2-7a44-4815-8420-cb6ed9f2e3d7" containerName="maas-api" Nov 02 18:36:26.547276 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.547250 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.557419 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.557390 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt"] Nov 02 18:36:26.647899 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.647857 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0659edff-3cd9-495d-bbce-c19f48452bf4-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.648097 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.647912 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0659edff-3cd9-495d-bbce-c19f48452bf4-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.648097 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.647985 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0659edff-3cd9-495d-bbce-c19f48452bf4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.648097 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.648040 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0659edff-3cd9-495d-bbce-c19f48452bf4-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.648097 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.648093 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0659edff-3cd9-495d-bbce-c19f48452bf4-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.648276 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.648124 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0659edff-3cd9-495d-bbce-c19f48452bf4-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.648276 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.648140 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6lll\" (UniqueName: \"kubernetes.io/projected/0659edff-3cd9-495d-bbce-c19f48452bf4-kube-api-access-n6lll\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.648276 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.648236 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0659edff-3cd9-495d-bbce-c19f48452bf4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.648389 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.648284 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0659edff-3cd9-495d-bbce-c19f48452bf4-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.749007 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.748969 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0659edff-3cd9-495d-bbce-c19f48452bf4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.749216 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.749034 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0659edff-3cd9-495d-bbce-c19f48452bf4-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.749216 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.749068 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0659edff-3cd9-495d-bbce-c19f48452bf4-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.749216 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.749099 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0659edff-3cd9-495d-bbce-c19f48452bf4-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.749216 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.749120 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0659edff-3cd9-495d-bbce-c19f48452bf4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.749216 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.749161 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0659edff-3cd9-495d-bbce-c19f48452bf4-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.749216 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.749187 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0659edff-3cd9-495d-bbce-c19f48452bf4-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.749561 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.749226 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0659edff-3cd9-495d-bbce-c19f48452bf4-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.749561 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.749247 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6lll\" (UniqueName: \"kubernetes.io/projected/0659edff-3cd9-495d-bbce-c19f48452bf4-kube-api-access-n6lll\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.749561 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.749479 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0659edff-3cd9-495d-bbce-c19f48452bf4-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.749780 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.749760 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0659edff-3cd9-495d-bbce-c19f48452bf4-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.749832 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.749803 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0659edff-3cd9-495d-bbce-c19f48452bf4-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.749891 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.749868 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0659edff-3cd9-495d-bbce-c19f48452bf4-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.749951 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.749933 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0659edff-3cd9-495d-bbce-c19f48452bf4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.751547 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.751525 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0659edff-3cd9-495d-bbce-c19f48452bf4-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.751822 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.751804 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0659edff-3cd9-495d-bbce-c19f48452bf4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.757046 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.757018 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0659edff-3cd9-495d-bbce-c19f48452bf4-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.757359 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.757337 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6lll\" (UniqueName: \"kubernetes.io/projected/0659edff-3cd9-495d-bbce-c19f48452bf4-kube-api-access-n6lll\") pod \"maas-default-gateway-openshift-default-65c448b547-th4lt\" (UID: \"0659edff-3cd9-495d-bbce-c19f48452bf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:26.864458 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:26.864364 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:27.143041 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:27.143013 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt"] Nov 02 18:36:27.145396 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:36:27.145367 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0659edff_3cd9_495d_bbce_c19f48452bf4.slice/crio-05c6fed934ec1d488183b2bf802638fdbd7d672c762eb38ea6a24f96bc1691f4 WatchSource:0}: Error finding container 05c6fed934ec1d488183b2bf802638fdbd7d672c762eb38ea6a24f96bc1691f4: Status 404 returned error can't find the container with id 05c6fed934ec1d488183b2bf802638fdbd7d672c762eb38ea6a24f96bc1691f4 Nov 02 18:36:27.147704 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:27.147665 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:36:27.147787 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:27.147760 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:36:27.147826 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:27.147807 5601 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 18:36:27.475216 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:27.475179 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" event={"ID":"0659edff-3cd9-495d-bbce-c19f48452bf4","Type":"ContainerStarted","Data":"b1b34248702d629f7427618a1d786a3282e9ac32378705af0b97617cd5b130e8"} Nov 02 18:36:27.475397 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:27.475233 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" event={"ID":"0659edff-3cd9-495d-bbce-c19f48452bf4","Type":"ContainerStarted","Data":"05c6fed934ec1d488183b2bf802638fdbd7d672c762eb38ea6a24f96bc1691f4"} Nov 02 18:36:27.497093 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:27.497040 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" podStartSLOduration=1.497023738 podStartE2EDuration="1.497023738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:36:27.496337989 +0000 UTC m=+3752.187392003" watchObservedRunningTime="2026-11-02 18:36:27.497023738 +0000 UTC m=+3752.188077744" Nov 02 18:36:27.864971 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:27.864882 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:28.866983 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:28.866948 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-th4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 02 18:36:28.867412 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:28.867014 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" podUID="0659edff-3cd9-495d-bbce-c19f48452bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 02 18:36:29.865176 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:29.865139 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-th4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 02 18:36:29.865362 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:29.865192 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" podUID="0659edff-3cd9-495d-bbce-c19f48452bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 02 18:36:30.865832 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:30.865793 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-th4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 02 18:36:30.866247 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:30.865852 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" podUID="0659edff-3cd9-495d-bbce-c19f48452bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 02 18:36:31.865164 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:31.865123 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-th4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 02 18:36:31.865358 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:31.865182 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" podUID="0659edff-3cd9-495d-bbce-c19f48452bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 02 18:36:32.865832 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:32.865789 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-th4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 02 18:36:32.866328 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:32.865872 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" podUID="0659edff-3cd9-495d-bbce-c19f48452bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 02 18:36:33.865107 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:33.865061 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-th4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 02 18:36:33.865391 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:33.865125 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" podUID="0659edff-3cd9-495d-bbce-c19f48452bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 02 18:36:34.483547 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:36:34.483507 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:36:35.866293 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:35.866205 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-th4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:36:35.866293 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:35.866267 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" podUID="0659edff-3cd9-495d-bbce-c19f48452bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:36:36.907809 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:36.907765 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-th4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:36:36.908217 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:36.907825 5601 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" podUID="0659edff-3cd9-495d-bbce-c19f48452bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:36:37.371354 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:37.371321 5601 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:37.371705 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:37.371683 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:36:37.372359 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:36:37.372342 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" Nov 02 18:37:04.484547 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:37:04.484506 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:37:34.483373 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:37:34.483333 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:37:46.794660 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:37:46.793749 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-jv8f4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:37:46.794660 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:37:46.793837 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" podUID="c32fa6f3-9665-4b3b-b093-22435190829b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:37:46.999641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:37:46.998735 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-wtks5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:37:46.999641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:37:46.998814 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-wtks5" podUID="63f70366-70f1-475a-8e45-eb6da6b4828e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:38:04.484154 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:38:04.484115 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:38:34.483919 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:38:34.483880 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:39:00.774589 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:00.774548 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b"] Nov 02 18:39:00.777933 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:00.777909 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b" Nov 02 18:39:00.784332 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:00.784302 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:39:00.784486 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:00.784380 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:39:00.785773 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:00.785637 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b"] Nov 02 18:39:00.856059 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:00.856020 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n9cp\" (UniqueName: \"kubernetes.io/projected/bc835e40-7052-455d-9c18-f769ec2cccd7-kube-api-access-2n9cp\") pod \"maas-api-revoke-sub-ddccbc82-qzz9b\" (UID: \"bc835e40-7052-455d-9c18-f769ec2cccd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b" Nov 02 18:39:00.856059 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:00.856062 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc835e40-7052-455d-9c18-f769ec2cccd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddccbc82-qzz9b\" (UID: \"bc835e40-7052-455d-9c18-f769ec2cccd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b" Nov 02 18:39:00.856306 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:00.856079 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc835e40-7052-455d-9c18-f769ec2cccd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddccbc82-qzz9b\" (UID: \"bc835e40-7052-455d-9c18-f769ec2cccd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b" Nov 02 18:39:00.957233 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:00.957200 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n9cp\" (UniqueName: \"kubernetes.io/projected/bc835e40-7052-455d-9c18-f769ec2cccd7-kube-api-access-2n9cp\") pod \"maas-api-revoke-sub-ddccbc82-qzz9b\" (UID: \"bc835e40-7052-455d-9c18-f769ec2cccd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b" Nov 02 18:39:00.957233 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:00.957236 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc835e40-7052-455d-9c18-f769ec2cccd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddccbc82-qzz9b\" (UID: \"bc835e40-7052-455d-9c18-f769ec2cccd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b" Nov 02 18:39:00.957460 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:00.957258 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc835e40-7052-455d-9c18-f769ec2cccd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddccbc82-qzz9b\" (UID: \"bc835e40-7052-455d-9c18-f769ec2cccd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b" Nov 02 18:39:00.957896 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:00.957876 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc835e40-7052-455d-9c18-f769ec2cccd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddccbc82-qzz9b\" (UID: \"bc835e40-7052-455d-9c18-f769ec2cccd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b" Nov 02 18:39:00.959605 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:00.959586 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc835e40-7052-455d-9c18-f769ec2cccd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddccbc82-qzz9b\" (UID: \"bc835e40-7052-455d-9c18-f769ec2cccd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b" Nov 02 18:39:00.967125 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:00.967096 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n9cp\" (UniqueName: \"kubernetes.io/projected/bc835e40-7052-455d-9c18-f769ec2cccd7-kube-api-access-2n9cp\") pod \"maas-api-revoke-sub-ddccbc82-qzz9b\" (UID: \"bc835e40-7052-455d-9c18-f769ec2cccd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b" Nov 02 18:39:01.092941 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:01.092854 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b" Nov 02 18:39:01.365296 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:01.365268 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b"] Nov 02 18:39:01.367400 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:39:01.367369 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc835e40_7052_455d_9c18_f769ec2cccd7.slice/crio-9b12b96779e948f46ce46f40d4cf0b5fb48e1efb0325834bc20195717cf818b6 WatchSource:0}: Error finding container 9b12b96779e948f46ce46f40d4cf0b5fb48e1efb0325834bc20195717cf818b6: Status 404 returned error can't find the container with id 9b12b96779e948f46ce46f40d4cf0b5fb48e1efb0325834bc20195717cf818b6 Nov 02 18:39:01.396107 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:01.395872 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b" event={"ID":"bc835e40-7052-455d-9c18-f769ec2cccd7","Type":"ContainerStarted","Data":"9b12b96779e948f46ce46f40d4cf0b5fb48e1efb0325834bc20195717cf818b6"} Nov 02 18:39:02.402728 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:02.402690 5601 generic.go:356] "Generic (PLEG): container finished" podID="bc835e40-7052-455d-9c18-f769ec2cccd7" containerID="66500781f2e6130147d5f56b882c023af2f6f65e5432fd757d068d6661824b23" exitCode=0 Nov 02 18:39:02.403174 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:02.402777 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b" event={"ID":"bc835e40-7052-455d-9c18-f769ec2cccd7","Type":"ContainerDied","Data":"66500781f2e6130147d5f56b882c023af2f6f65e5432fd757d068d6661824b23"} Nov 02 18:39:03.539053 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:03.539021 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b" Nov 02 18:39:03.579518 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:03.579486 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc835e40-7052-455d-9c18-f769ec2cccd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc835e40-7052-455d-9c18-f769ec2cccd7-maas-api-openshift-ca\") pod \"bc835e40-7052-455d-9c18-f769ec2cccd7\" (UID: \"bc835e40-7052-455d-9c18-f769ec2cccd7\") " Nov 02 18:39:03.579729 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:03.579627 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc835e40-7052-455d-9c18-f769ec2cccd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc835e40-7052-455d-9c18-f769ec2cccd7-maas-api-xks-ca\") pod \"bc835e40-7052-455d-9c18-f769ec2cccd7\" (UID: \"bc835e40-7052-455d-9c18-f769ec2cccd7\") " Nov 02 18:39:03.579729 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:03.579658 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc835e40-7052-455d-9c18-f769ec2cccd7-kube-api-access-2n9cp\" (UniqueName: \"kubernetes.io/projected/bc835e40-7052-455d-9c18-f769ec2cccd7-kube-api-access-2n9cp\") pod \"bc835e40-7052-455d-9c18-f769ec2cccd7\" (UID: \"bc835e40-7052-455d-9c18-f769ec2cccd7\") " Nov 02 18:39:03.579919 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:03.579892 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc835e40-7052-455d-9c18-f769ec2cccd7-maas-api-openshift-ca" pod "bc835e40-7052-455d-9c18-f769ec2cccd7" (UID: "bc835e40-7052-455d-9c18-f769ec2cccd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:39:03.581862 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:03.581838 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc835e40-7052-455d-9c18-f769ec2cccd7-kube-api-access-2n9cp" pod "bc835e40-7052-455d-9c18-f769ec2cccd7" (UID: "bc835e40-7052-455d-9c18-f769ec2cccd7"). InnerVolumeSpecName "kube-api-access-2n9cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:39:03.581862 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:03.581849 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc835e40-7052-455d-9c18-f769ec2cccd7-maas-api-xks-ca" pod "bc835e40-7052-455d-9c18-f769ec2cccd7" (UID: "bc835e40-7052-455d-9c18-f769ec2cccd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:39:03.680703 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:03.680666 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n9cp\" (UniqueName: \"kubernetes.io/projected/bc835e40-7052-455d-9c18-f769ec2cccd7-kube-api-access-2n9cp\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:39:03.680703 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:03.680699 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc835e40-7052-455d-9c18-f769ec2cccd7-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:39:03.680909 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:03.680714 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc835e40-7052-455d-9c18-f769ec2cccd7-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:39:04.413558 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:04.413524 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b" Nov 02 18:39:04.413757 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:04.413528 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b" event={"ID":"bc835e40-7052-455d-9c18-f769ec2cccd7","Type":"ContainerDied","Data":"9b12b96779e948f46ce46f40d4cf0b5fb48e1efb0325834bc20195717cf818b6"} Nov 02 18:39:04.413757 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:04.413660 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b12b96779e948f46ce46f40d4cf0b5fb48e1efb0325834bc20195717cf818b6" Nov 02 18:39:04.483090 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:39:04.483056 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:39:16.275427 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:16.275383 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn"] Nov 02 18:39:16.276007 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:16.275981 5601 cpu_manager.go:414] "containerMap: removing container" podUID="bc835e40-7052-455d-9c18-f769ec2cccd7" containerName="revoke-keys" Nov 02 18:39:16.276007 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:16.276001 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc835e40-7052-455d-9c18-f769ec2cccd7" containerName="revoke-keys" Nov 02 18:39:16.276153 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:16.276131 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc835e40-7052-455d-9c18-f769ec2cccd7" containerName="revoke-keys" Nov 02 18:39:16.279002 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:16.278977 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn" Nov 02 18:39:16.322661 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:16.314982 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:39:16.322661 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:16.315123 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:39:16.322661 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:16.319770 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn"] Nov 02 18:39:16.327156 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:16.327127 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0cc2be-8abf-449a-a23e-73428c782cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2aafe005-x65pn\" (UID: \"3b0cc2be-8abf-449a-a23e-73428c782cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn" Nov 02 18:39:16.327482 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:16.327461 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4hc9\" (UniqueName: \"kubernetes.io/projected/3b0cc2be-8abf-449a-a23e-73428c782cd3-kube-api-access-c4hc9\") pod \"maas-api-revoke-sub-2aafe005-x65pn\" (UID: \"3b0cc2be-8abf-449a-a23e-73428c782cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn" Nov 02 18:39:16.327641 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:16.327597 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0cc2be-8abf-449a-a23e-73428c782cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2aafe005-x65pn\" (UID: \"3b0cc2be-8abf-449a-a23e-73428c782cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn" Nov 02 18:39:16.428611 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:16.428576 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4hc9\" (UniqueName: \"kubernetes.io/projected/3b0cc2be-8abf-449a-a23e-73428c782cd3-kube-api-access-c4hc9\") pod \"maas-api-revoke-sub-2aafe005-x65pn\" (UID: \"3b0cc2be-8abf-449a-a23e-73428c782cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn" Nov 02 18:39:16.428611 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:16.428644 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0cc2be-8abf-449a-a23e-73428c782cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2aafe005-x65pn\" (UID: \"3b0cc2be-8abf-449a-a23e-73428c782cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn" Nov 02 18:39:16.428896 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:16.428729 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0cc2be-8abf-449a-a23e-73428c782cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2aafe005-x65pn\" (UID: \"3b0cc2be-8abf-449a-a23e-73428c782cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn" Nov 02 18:39:16.429517 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:16.429480 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0cc2be-8abf-449a-a23e-73428c782cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2aafe005-x65pn\" (UID: \"3b0cc2be-8abf-449a-a23e-73428c782cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn" Nov 02 18:39:16.431308 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:16.431275 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0cc2be-8abf-449a-a23e-73428c782cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2aafe005-x65pn\" (UID: \"3b0cc2be-8abf-449a-a23e-73428c782cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn" Nov 02 18:39:16.437569 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:16.437540 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4hc9\" (UniqueName: \"kubernetes.io/projected/3b0cc2be-8abf-449a-a23e-73428c782cd3-kube-api-access-c4hc9\") pod \"maas-api-revoke-sub-2aafe005-x65pn\" (UID: \"3b0cc2be-8abf-449a-a23e-73428c782cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn" Nov 02 18:39:16.641260 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:16.641163 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn" Nov 02 18:39:16.930154 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:16.930122 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn"] Nov 02 18:39:16.933196 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:39:16.933163 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0cc2be_8abf_449a_a23e_73428c782cd3.slice/crio-fc94964e7782ffb65328e5c898fcf23e961346fdd01463d6861d1bbea17a3006 WatchSource:0}: Error finding container fc94964e7782ffb65328e5c898fcf23e961346fdd01463d6861d1bbea17a3006: Status 404 returned error can't find the container with id fc94964e7782ffb65328e5c898fcf23e961346fdd01463d6861d1bbea17a3006 Nov 02 18:39:17.512446 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:17.512404 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn" event={"ID":"3b0cc2be-8abf-449a-a23e-73428c782cd3","Type":"ContainerStarted","Data":"2547ebbecb6c7e3e71c20369110c29632312c52aea294a28e116c2dfd7a51da9"} Nov 02 18:39:17.513947 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:17.513920 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn" event={"ID":"3b0cc2be-8abf-449a-a23e-73428c782cd3","Type":"ContainerStarted","Data":"fc94964e7782ffb65328e5c898fcf23e961346fdd01463d6861d1bbea17a3006"} Nov 02 18:39:17.535450 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:17.535217 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn" podStartSLOduration=1.535199844 podStartE2EDuration="1.535199844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:39:17.53220519 +0000 UTC m=+3922.223259196" watchObservedRunningTime="2026-11-02 18:39:17.535199844 +0000 UTC m=+3922.226253860" Nov 02 18:39:18.519636 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:18.519581 5601 generic.go:356] "Generic (PLEG): container finished" podID="3b0cc2be-8abf-449a-a23e-73428c782cd3" containerID="2547ebbecb6c7e3e71c20369110c29632312c52aea294a28e116c2dfd7a51da9" exitCode=0 Nov 02 18:39:18.520108 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:18.519647 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn" event={"ID":"3b0cc2be-8abf-449a-a23e-73428c782cd3","Type":"ContainerDied","Data":"2547ebbecb6c7e3e71c20369110c29632312c52aea294a28e116c2dfd7a51da9"} Nov 02 18:39:19.655530 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:19.655505 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn" Nov 02 18:39:19.824909 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:19.824828 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0cc2be-8abf-449a-a23e-73428c782cd3-kube-api-access-c4hc9\" (UniqueName: \"kubernetes.io/projected/3b0cc2be-8abf-449a-a23e-73428c782cd3-kube-api-access-c4hc9\") pod \"3b0cc2be-8abf-449a-a23e-73428c782cd3\" (UID: \"3b0cc2be-8abf-449a-a23e-73428c782cd3\") " Nov 02 18:39:19.824909 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:19.824886 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b0cc2be-8abf-449a-a23e-73428c782cd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0cc2be-8abf-449a-a23e-73428c782cd3-maas-api-xks-ca\") pod \"3b0cc2be-8abf-449a-a23e-73428c782cd3\" (UID: \"3b0cc2be-8abf-449a-a23e-73428c782cd3\") " Nov 02 18:39:19.825111 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:19.824971 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b0cc2be-8abf-449a-a23e-73428c782cd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0cc2be-8abf-449a-a23e-73428c782cd3-maas-api-openshift-ca\") pod \"3b0cc2be-8abf-449a-a23e-73428c782cd3\" (UID: \"3b0cc2be-8abf-449a-a23e-73428c782cd3\") " Nov 02 18:39:19.825342 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:19.825314 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0cc2be-8abf-449a-a23e-73428c782cd3-maas-api-openshift-ca" pod "3b0cc2be-8abf-449a-a23e-73428c782cd3" (UID: "3b0cc2be-8abf-449a-a23e-73428c782cd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:39:19.827101 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:19.827071 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b0cc2be-8abf-449a-a23e-73428c782cd3-maas-api-xks-ca" pod "3b0cc2be-8abf-449a-a23e-73428c782cd3" (UID: "3b0cc2be-8abf-449a-a23e-73428c782cd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:39:19.827201 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:19.827095 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0cc2be-8abf-449a-a23e-73428c782cd3-kube-api-access-c4hc9" pod "3b0cc2be-8abf-449a-a23e-73428c782cd3" (UID: "3b0cc2be-8abf-449a-a23e-73428c782cd3"). InnerVolumeSpecName "kube-api-access-c4hc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:39:19.926262 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:19.926229 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4hc9\" (UniqueName: \"kubernetes.io/projected/3b0cc2be-8abf-449a-a23e-73428c782cd3-kube-api-access-c4hc9\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:39:19.926262 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:19.926260 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0cc2be-8abf-449a-a23e-73428c782cd3-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:39:19.926451 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:19.926270 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0cc2be-8abf-449a-a23e-73428c782cd3-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:39:20.529663 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:20.529609 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn" Nov 02 18:39:20.529847 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:20.529663 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn" event={"ID":"3b0cc2be-8abf-449a-a23e-73428c782cd3","Type":"ContainerDied","Data":"fc94964e7782ffb65328e5c898fcf23e961346fdd01463d6861d1bbea17a3006"} Nov 02 18:39:20.529847 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:39:20.529718 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc94964e7782ffb65328e5c898fcf23e961346fdd01463d6861d1bbea17a3006" Nov 02 18:39:34.482993 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:39:34.482955 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:40:04.483157 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:40:04.483121 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:40:34.494221 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:40:34.494180 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:41:04.484127 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:41:04.484083 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:41:34.527702 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:41:34.527663 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:42:04.483831 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:42:04.483747 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:42:08.415839 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:42:08.415791 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-th4lt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:42:08.416289 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:42:08.415868 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" podUID="0659edff-3cd9-495d-bbce-c19f48452bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:42:34.525060 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:42:34.522118 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:43:04.483759 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:43:04.483717 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:43:34.551888 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:43:34.551829 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:44:04.025235 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:44:04.025195 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b"] Nov 02 18:44:04.027987 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:44:04.027807 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddccbc82-qzz9b"] Nov 02 18:44:04.484268 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:44:04.484232 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:44:05.424809 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:44:05.424773 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc835e40-7052-455d-9c18-f769ec2cccd7" path="/var/lib/kubelet/pods/bc835e40-7052-455d-9c18-f769ec2cccd7/volumes" Nov 02 18:44:20.021286 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:44:20.021246 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn"] Nov 02 18:44:20.024193 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:44:20.024163 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aafe005-x65pn"] Nov 02 18:44:21.424683 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:44:21.424646 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0cc2be-8abf-449a-a23e-73428c782cd3" path="/var/lib/kubelet/pods/3b0cc2be-8abf-449a-a23e-73428c782cd3/volumes" Nov 02 18:44:34.483495 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:44:34.483455 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:45:04.485339 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:45:04.485254 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:45:12.318484 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:12.318441 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d"] Nov 02 18:45:12.319016 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:12.318839 5601 cpu_manager.go:414] "containerMap: removing container" podUID="3b0cc2be-8abf-449a-a23e-73428c782cd3" containerName="revoke-keys" Nov 02 18:45:12.319016 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:12.318851 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0cc2be-8abf-449a-a23e-73428c782cd3" containerName="revoke-keys" Nov 02 18:45:12.319016 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:12.318937 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0cc2be-8abf-449a-a23e-73428c782cd3" containerName="revoke-keys" Nov 02 18:45:12.321830 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:12.321813 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d" Nov 02 18:45:12.327526 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:12.327492 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:45:12.327698 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:12.327527 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:45:12.328438 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:12.328414 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d"] Nov 02 18:45:12.447699 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:12.447668 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec18c771-769f-4d08-a524-1b240e85b470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-jqt7d\" (UID: \"ec18c771-769f-4d08-a524-1b240e85b470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d" Nov 02 18:45:12.447870 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:12.447720 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec18c771-769f-4d08-a524-1b240e85b470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-jqt7d\" (UID: \"ec18c771-769f-4d08-a524-1b240e85b470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d" Nov 02 18:45:12.447870 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:12.447812 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44l5n\" (UniqueName: \"kubernetes.io/projected/ec18c771-769f-4d08-a524-1b240e85b470-kube-api-access-44l5n\") pod \"maas-api-revoke-sub-e2547622-jqt7d\" (UID: \"ec18c771-769f-4d08-a524-1b240e85b470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d" Nov 02 18:45:12.548324 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:12.548290 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44l5n\" (UniqueName: \"kubernetes.io/projected/ec18c771-769f-4d08-a524-1b240e85b470-kube-api-access-44l5n\") pod \"maas-api-revoke-sub-e2547622-jqt7d\" (UID: \"ec18c771-769f-4d08-a524-1b240e85b470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d" Nov 02 18:45:12.548538 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:12.548394 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec18c771-769f-4d08-a524-1b240e85b470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-jqt7d\" (UID: \"ec18c771-769f-4d08-a524-1b240e85b470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d" Nov 02 18:45:12.548538 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:12.548452 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec18c771-769f-4d08-a524-1b240e85b470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-jqt7d\" (UID: \"ec18c771-769f-4d08-a524-1b240e85b470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d" Nov 02 18:45:12.549177 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:12.549149 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec18c771-769f-4d08-a524-1b240e85b470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-jqt7d\" (UID: \"ec18c771-769f-4d08-a524-1b240e85b470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d" Nov 02 18:45:12.550907 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:12.550882 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec18c771-769f-4d08-a524-1b240e85b470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-jqt7d\" (UID: \"ec18c771-769f-4d08-a524-1b240e85b470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d" Nov 02 18:45:12.556782 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:12.556759 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44l5n\" (UniqueName: \"kubernetes.io/projected/ec18c771-769f-4d08-a524-1b240e85b470-kube-api-access-44l5n\") pod \"maas-api-revoke-sub-e2547622-jqt7d\" (UID: \"ec18c771-769f-4d08-a524-1b240e85b470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d" Nov 02 18:45:12.636110 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:12.636023 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d" Nov 02 18:45:12.909351 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:12.909323 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d"] Nov 02 18:45:12.914901 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:45:12.914871 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec18c771_769f_4d08_a524_1b240e85b470.slice/crio-c29d1ce3bc19afb251b5d94d4891028b2a967025d3f5b1dd64bd4da4a3a481a1 WatchSource:0}: Error finding container c29d1ce3bc19afb251b5d94d4891028b2a967025d3f5b1dd64bd4da4a3a481a1: Status 404 returned error can't find the container with id c29d1ce3bc19afb251b5d94d4891028b2a967025d3f5b1dd64bd4da4a3a481a1 Nov 02 18:45:13.385044 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:13.384998 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d" event={"ID":"ec18c771-769f-4d08-a524-1b240e85b470","Type":"ContainerStarted","Data":"392a72e86a494519204d0557aebecf733d2c6d867c6efacfba984f158681cf9b"} Nov 02 18:45:13.385549 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:13.385073 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d" event={"ID":"ec18c771-769f-4d08-a524-1b240e85b470","Type":"ContainerStarted","Data":"c29d1ce3bc19afb251b5d94d4891028b2a967025d3f5b1dd64bd4da4a3a481a1"} Nov 02 18:45:13.403569 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:13.403501 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d" podStartSLOduration=1.403489058 podStartE2EDuration="1.403489058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:13.402483768 +0000 UTC m=+4278.093537781" watchObservedRunningTime="2026-11-02 18:45:13.403489058 +0000 UTC m=+4278.094543070" Nov 02 18:45:14.392042 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:14.392009 5601 generic.go:356] "Generic (PLEG): container finished" podID="ec18c771-769f-4d08-a524-1b240e85b470" containerID="392a72e86a494519204d0557aebecf733d2c6d867c6efacfba984f158681cf9b" exitCode=0 Nov 02 18:45:14.392423 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:14.392089 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d" event={"ID":"ec18c771-769f-4d08-a524-1b240e85b470","Type":"ContainerDied","Data":"392a72e86a494519204d0557aebecf733d2c6d867c6efacfba984f158681cf9b"} Nov 02 18:45:15.533466 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:15.533434 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d" Nov 02 18:45:15.578582 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:15.578550 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec18c771-769f-4d08-a524-1b240e85b470-kube-api-access-44l5n\" (UniqueName: \"kubernetes.io/projected/ec18c771-769f-4d08-a524-1b240e85b470-kube-api-access-44l5n\") pod \"ec18c771-769f-4d08-a524-1b240e85b470\" (UID: \"ec18c771-769f-4d08-a524-1b240e85b470\") " Nov 02 18:45:15.578735 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:15.578626 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec18c771-769f-4d08-a524-1b240e85b470-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec18c771-769f-4d08-a524-1b240e85b470-maas-api-openshift-ca\") pod \"ec18c771-769f-4d08-a524-1b240e85b470\" (UID: \"ec18c771-769f-4d08-a524-1b240e85b470\") " Nov 02 18:45:15.578735 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:15.578670 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec18c771-769f-4d08-a524-1b240e85b470-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec18c771-769f-4d08-a524-1b240e85b470-maas-api-xks-ca\") pod \"ec18c771-769f-4d08-a524-1b240e85b470\" (UID: \"ec18c771-769f-4d08-a524-1b240e85b470\") " Nov 02 18:45:15.578934 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:15.578911 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec18c771-769f-4d08-a524-1b240e85b470-maas-api-openshift-ca" pod "ec18c771-769f-4d08-a524-1b240e85b470" (UID: "ec18c771-769f-4d08-a524-1b240e85b470"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:45:15.580714 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:15.580690 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec18c771-769f-4d08-a524-1b240e85b470-maas-api-xks-ca" pod "ec18c771-769f-4d08-a524-1b240e85b470" (UID: "ec18c771-769f-4d08-a524-1b240e85b470"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:45:15.580843 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:15.580822 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec18c771-769f-4d08-a524-1b240e85b470-kube-api-access-44l5n" pod "ec18c771-769f-4d08-a524-1b240e85b470" (UID: "ec18c771-769f-4d08-a524-1b240e85b470"). InnerVolumeSpecName "kube-api-access-44l5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:45:15.680112 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:15.680086 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44l5n\" (UniqueName: \"kubernetes.io/projected/ec18c771-769f-4d08-a524-1b240e85b470-kube-api-access-44l5n\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:45:15.680268 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:15.680114 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec18c771-769f-4d08-a524-1b240e85b470-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:45:15.680268 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:15.680125 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec18c771-769f-4d08-a524-1b240e85b470-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:45:16.405419 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:16.405383 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d" Nov 02 18:45:16.405600 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:16.405413 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d" event={"ID":"ec18c771-769f-4d08-a524-1b240e85b470","Type":"ContainerDied","Data":"c29d1ce3bc19afb251b5d94d4891028b2a967025d3f5b1dd64bd4da4a3a481a1"} Nov 02 18:45:16.405600 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:16.405465 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c29d1ce3bc19afb251b5d94d4891028b2a967025d3f5b1dd64bd4da4a3a481a1" Nov 02 18:45:34.484305 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:45:34.484263 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:45:58.189699 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:58.189669 5601 scope.go:122] "RemoveContainer" containerID="2547ebbecb6c7e3e71c20369110c29632312c52aea294a28e116c2dfd7a51da9" Nov 02 18:45:58.198444 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:45:58.198422 5601 scope.go:122] "RemoveContainer" containerID="66500781f2e6130147d5f56b882c023af2f6f65e5432fd757d068d6661824b23" Nov 02 18:46:04.524017 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:46:04.523975 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:46:34.486986 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:46:34.486923 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:47:04.483299 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:47:04.483260 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:47:34.484608 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:47:34.484566 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:48:04.526120 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:48:04.526037 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:48:34.482562 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:48:34.482526 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:49:04.483199 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:49:04.483159 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:49:08.413904 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:49:08.413801 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-th4lt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:49:08.413904 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:49:08.413877 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" podUID="0659edff-3cd9-495d-bbce-c19f48452bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:49:34.482834 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:49:34.482792 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:50:04.483242 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:50:04.483204 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:50:16.019939 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:50:16.019893 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d"] Nov 02 18:50:16.022656 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:50:16.022611 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jqt7d"] Nov 02 18:50:17.424207 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:50:17.424158 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec18c771-769f-4d08-a524-1b240e85b470" path="/var/lib/kubelet/pods/ec18c771-769f-4d08-a524-1b240e85b470/volumes" Nov 02 18:50:34.482818 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:50:34.482774 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:51:04.484892 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:51:04.484801 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:51:23.417782 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:51:23.417738 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-th4lt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:51:23.418314 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:51:23.417794 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" podUID="0659edff-3cd9-495d-bbce-c19f48452bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:51:34.483946 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:51:34.483904 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:51:58.267544 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:51:58.267509 5601 scope.go:122] "RemoveContainer" containerID="392a72e86a494519204d0557aebecf733d2c6d867c6efacfba984f158681cf9b" Nov 02 18:52:04.484059 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:52:04.484020 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:52:34.484146 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:52:34.484104 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:53:04.484680 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:53:04.484641 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:53:34.486938 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:53:34.486887 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:53:37.976702 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:37.976596 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd"] Nov 02 18:53:37.977070 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:37.977014 5601 cpu_manager.go:414] "containerMap: removing container" podUID="ec18c771-769f-4d08-a524-1b240e85b470" containerName="revoke-keys" Nov 02 18:53:37.977070 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:37.977026 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec18c771-769f-4d08-a524-1b240e85b470" containerName="revoke-keys" Nov 02 18:53:37.977149 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:37.977098 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec18c771-769f-4d08-a524-1b240e85b470" containerName="revoke-keys" Nov 02 18:53:37.979092 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:37.979075 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd" Nov 02 18:53:37.987240 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:37.986756 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:53:37.987240 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:37.986879 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:53:37.987611 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:37.987584 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd"] Nov 02 18:53:38.105861 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:38.105817 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c1e0b7-7e96-4e49-a386-3407ad293494-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1700323-vt2dd\" (UID: \"e9c1e0b7-7e96-4e49-a386-3407ad293494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd" Nov 02 18:53:38.105861 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:38.105864 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c1e0b7-7e96-4e49-a386-3407ad293494-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1700323-vt2dd\" (UID: \"e9c1e0b7-7e96-4e49-a386-3407ad293494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd" Nov 02 18:53:38.106087 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:38.105882 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8zjb\" (UniqueName: \"kubernetes.io/projected/e9c1e0b7-7e96-4e49-a386-3407ad293494-kube-api-access-c8zjb\") pod \"maas-api-revoke-sub-f1700323-vt2dd\" (UID: \"e9c1e0b7-7e96-4e49-a386-3407ad293494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd" Nov 02 18:53:38.207486 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:38.207442 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c1e0b7-7e96-4e49-a386-3407ad293494-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1700323-vt2dd\" (UID: \"e9c1e0b7-7e96-4e49-a386-3407ad293494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd" Nov 02 18:53:38.207730 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:38.207494 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c1e0b7-7e96-4e49-a386-3407ad293494-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1700323-vt2dd\" (UID: \"e9c1e0b7-7e96-4e49-a386-3407ad293494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd" Nov 02 18:53:38.207730 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:38.207523 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8zjb\" (UniqueName: \"kubernetes.io/projected/e9c1e0b7-7e96-4e49-a386-3407ad293494-kube-api-access-c8zjb\") pod \"maas-api-revoke-sub-f1700323-vt2dd\" (UID: \"e9c1e0b7-7e96-4e49-a386-3407ad293494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd" Nov 02 18:53:38.208202 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:38.208176 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c1e0b7-7e96-4e49-a386-3407ad293494-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1700323-vt2dd\" (UID: \"e9c1e0b7-7e96-4e49-a386-3407ad293494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd" Nov 02 18:53:38.210017 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:38.209992 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c1e0b7-7e96-4e49-a386-3407ad293494-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1700323-vt2dd\" (UID: \"e9c1e0b7-7e96-4e49-a386-3407ad293494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd" Nov 02 18:53:38.215515 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:38.215496 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8zjb\" (UniqueName: \"kubernetes.io/projected/e9c1e0b7-7e96-4e49-a386-3407ad293494-kube-api-access-c8zjb\") pod \"maas-api-revoke-sub-f1700323-vt2dd\" (UID: \"e9c1e0b7-7e96-4e49-a386-3407ad293494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd" Nov 02 18:53:38.294165 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:38.294086 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd" Nov 02 18:53:38.564336 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:38.564307 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd"] Nov 02 18:53:38.567126 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:53:38.567091 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c1e0b7_7e96_4e49_a386_3407ad293494.slice/crio-b9d870797193f161df4f7288996266aced1c597b2eea711c29a090e0af0b086b WatchSource:0}: Error finding container b9d870797193f161df4f7288996266aced1c597b2eea711c29a090e0af0b086b: Status 404 returned error can't find the container with id b9d870797193f161df4f7288996266aced1c597b2eea711c29a090e0af0b086b Nov 02 18:53:39.303830 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:39.303742 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd" event={"ID":"e9c1e0b7-7e96-4e49-a386-3407ad293494","Type":"ContainerStarted","Data":"e1d93eb2cf6ab937ddaf2dbca02c0d693adfc41f877ba8d936be20f0b55e4b4f"} Nov 02 18:53:39.304306 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:39.303839 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd" event={"ID":"e9c1e0b7-7e96-4e49-a386-3407ad293494","Type":"ContainerStarted","Data":"b9d870797193f161df4f7288996266aced1c597b2eea711c29a090e0af0b086b"} Nov 02 18:53:39.322104 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:39.322056 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd" podStartSLOduration=2.322044648 podStartE2EDuration="2.322044648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:53:39.320093961 +0000 UTC m=+4784.011147973" watchObservedRunningTime="2026-11-02 18:53:39.322044648 +0000 UTC m=+4784.013098739" Nov 02 18:53:40.309786 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:40.309743 5601 generic.go:356] "Generic (PLEG): container finished" podID="e9c1e0b7-7e96-4e49-a386-3407ad293494" containerID="e1d93eb2cf6ab937ddaf2dbca02c0d693adfc41f877ba8d936be20f0b55e4b4f" exitCode=0 Nov 02 18:53:40.310296 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:40.309838 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd" event={"ID":"e9c1e0b7-7e96-4e49-a386-3407ad293494","Type":"ContainerDied","Data":"e1d93eb2cf6ab937ddaf2dbca02c0d693adfc41f877ba8d936be20f0b55e4b4f"} Nov 02 18:53:41.446358 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:41.446332 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd" Nov 02 18:53:41.538471 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:41.538437 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9c1e0b7-7e96-4e49-a386-3407ad293494-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c1e0b7-7e96-4e49-a386-3407ad293494-maas-api-xks-ca\") pod \"e9c1e0b7-7e96-4e49-a386-3407ad293494\" (UID: \"e9c1e0b7-7e96-4e49-a386-3407ad293494\") " Nov 02 18:53:41.538687 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:41.538507 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c1e0b7-7e96-4e49-a386-3407ad293494-kube-api-access-c8zjb\" (UniqueName: \"kubernetes.io/projected/e9c1e0b7-7e96-4e49-a386-3407ad293494-kube-api-access-c8zjb\") pod \"e9c1e0b7-7e96-4e49-a386-3407ad293494\" (UID: \"e9c1e0b7-7e96-4e49-a386-3407ad293494\") " Nov 02 18:53:41.538687 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:41.538572 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c1e0b7-7e96-4e49-a386-3407ad293494-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c1e0b7-7e96-4e49-a386-3407ad293494-maas-api-openshift-ca\") pod \"e9c1e0b7-7e96-4e49-a386-3407ad293494\" (UID: \"e9c1e0b7-7e96-4e49-a386-3407ad293494\") " Nov 02 18:53:41.538939 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:41.538916 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c1e0b7-7e96-4e49-a386-3407ad293494-maas-api-openshift-ca" pod "e9c1e0b7-7e96-4e49-a386-3407ad293494" (UID: "e9c1e0b7-7e96-4e49-a386-3407ad293494"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:53:41.540510 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:41.540488 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c1e0b7-7e96-4e49-a386-3407ad293494-maas-api-xks-ca" pod "e9c1e0b7-7e96-4e49-a386-3407ad293494" (UID: "e9c1e0b7-7e96-4e49-a386-3407ad293494"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:53:41.540645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:41.540611 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c1e0b7-7e96-4e49-a386-3407ad293494-kube-api-access-c8zjb" pod "e9c1e0b7-7e96-4e49-a386-3407ad293494" (UID: "e9c1e0b7-7e96-4e49-a386-3407ad293494"). InnerVolumeSpecName "kube-api-access-c8zjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:53:41.640120 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:41.640049 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c1e0b7-7e96-4e49-a386-3407ad293494-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:53:41.640120 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:41.640080 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8zjb\" (UniqueName: \"kubernetes.io/projected/e9c1e0b7-7e96-4e49-a386-3407ad293494-kube-api-access-c8zjb\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:53:41.640120 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:41.640092 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c1e0b7-7e96-4e49-a386-3407ad293494-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:53:42.321087 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:42.321051 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd" Nov 02 18:53:42.321087 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:42.321077 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd" event={"ID":"e9c1e0b7-7e96-4e49-a386-3407ad293494","Type":"ContainerDied","Data":"b9d870797193f161df4f7288996266aced1c597b2eea711c29a090e0af0b086b"} Nov 02 18:53:42.321316 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:42.321127 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9d870797193f161df4f7288996266aced1c597b2eea711c29a090e0af0b086b" Nov 02 18:53:48.381243 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:48.381204 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd"] Nov 02 18:53:48.381655 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:48.381579 5601 cpu_manager.go:414] "containerMap: removing container" podUID="e9c1e0b7-7e96-4e49-a386-3407ad293494" containerName="revoke-keys" Nov 02 18:53:48.381655 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:48.381590 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c1e0b7-7e96-4e49-a386-3407ad293494" containerName="revoke-keys" Nov 02 18:53:48.381767 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:48.381686 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c1e0b7-7e96-4e49-a386-3407ad293494" containerName="revoke-keys" Nov 02 18:53:48.385010 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:48.384993 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd" Nov 02 18:53:48.390451 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:48.390425 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:53:48.390595 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:48.390559 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:53:48.391504 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:48.391477 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd"] Nov 02 18:53:48.502114 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:48.502074 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-418bf626-n8lpd\" (UID: \"9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd" Nov 02 18:53:48.502114 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:48.502114 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg5zl\" (UniqueName: \"kubernetes.io/projected/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-kube-api-access-tg5zl\") pod \"maas-api-revoke-sub-418bf626-n8lpd\" (UID: \"9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd" Nov 02 18:53:48.502333 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:48.502210 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-418bf626-n8lpd\" (UID: \"9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd" Nov 02 18:53:48.603038 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:48.603003 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg5zl\" (UniqueName: \"kubernetes.io/projected/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-kube-api-access-tg5zl\") pod \"maas-api-revoke-sub-418bf626-n8lpd\" (UID: \"9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd" Nov 02 18:53:48.603219 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:48.603060 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-418bf626-n8lpd\" (UID: \"9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd" Nov 02 18:53:48.603219 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:48.603146 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-418bf626-n8lpd\" (UID: \"9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd" Nov 02 18:53:48.603867 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:48.603841 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-418bf626-n8lpd\" (UID: \"9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd" Nov 02 18:53:48.605532 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:48.605510 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-418bf626-n8lpd\" (UID: \"9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd" Nov 02 18:53:48.610788 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:48.610762 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg5zl\" (UniqueName: \"kubernetes.io/projected/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-kube-api-access-tg5zl\") pod \"maas-api-revoke-sub-418bf626-n8lpd\" (UID: \"9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd" Nov 02 18:53:48.699498 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:48.699456 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd" Nov 02 18:53:48.975372 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:48.975291 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd"] Nov 02 18:53:48.978398 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:53:48.978369 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5a0e1f_4b06_4426_bf1a_abdbee7eb9e6.slice/crio-167e392727d993898c53993e6accaefa424605353b71ce682535fb26986182d3 WatchSource:0}: Error finding container 167e392727d993898c53993e6accaefa424605353b71ce682535fb26986182d3: Status 404 returned error can't find the container with id 167e392727d993898c53993e6accaefa424605353b71ce682535fb26986182d3 Nov 02 18:53:49.356070 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:49.356041 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd" event={"ID":"9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6","Type":"ContainerStarted","Data":"d86db00d7b6bb1a87b5669639260f60a20b6f65442498677f373a02b110ac8e7"} Nov 02 18:53:49.356227 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:49.356099 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd" event={"ID":"9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6","Type":"ContainerStarted","Data":"167e392727d993898c53993e6accaefa424605353b71ce682535fb26986182d3"} Nov 02 18:53:49.372958 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:49.372882 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd" podStartSLOduration=1.372867314 podStartE2EDuration="1.372867314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:53:49.37161232 +0000 UTC m=+4794.062666335" watchObservedRunningTime="2026-11-02 18:53:49.372867314 +0000 UTC m=+4794.063921349" Nov 02 18:53:50.362309 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:50.362271 5601 generic.go:356] "Generic (PLEG): container finished" podID="9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6" containerID="d86db00d7b6bb1a87b5669639260f60a20b6f65442498677f373a02b110ac8e7" exitCode=0 Nov 02 18:53:50.362730 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:50.362353 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd" event={"ID":"9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6","Type":"ContainerDied","Data":"d86db00d7b6bb1a87b5669639260f60a20b6f65442498677f373a02b110ac8e7"} Nov 02 18:53:51.498967 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:51.498943 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd" Nov 02 18:53:51.629727 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:51.629656 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-maas-api-xks-ca\") pod \"9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6\" (UID: \"9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6\") " Nov 02 18:53:51.629727 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:51.629697 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-kube-api-access-tg5zl\" (UniqueName: \"kubernetes.io/projected/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-kube-api-access-tg5zl\") pod \"9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6\" (UID: \"9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6\") " Nov 02 18:53:51.629910 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:51.629727 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-maas-api-openshift-ca\") pod \"9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6\" (UID: \"9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6\") " Nov 02 18:53:51.630105 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:51.630084 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-maas-api-openshift-ca" pod "9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6" (UID: "9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:53:51.631863 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:51.631837 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-maas-api-xks-ca" pod "9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6" (UID: "9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:53:51.631978 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:51.631862 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-kube-api-access-tg5zl" pod "9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6" (UID: "9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6"). InnerVolumeSpecName "kube-api-access-tg5zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:53:51.730535 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:51.730501 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:53:51.730706 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:51.730538 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg5zl\" (UniqueName: \"kubernetes.io/projected/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-kube-api-access-tg5zl\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:53:51.730706 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:51.730554 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:53:52.373026 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:52.372990 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd" Nov 02 18:53:52.373026 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:52.373002 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd" event={"ID":"9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6","Type":"ContainerDied","Data":"167e392727d993898c53993e6accaefa424605353b71ce682535fb26986182d3"} Nov 02 18:53:52.373299 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:53:52.373058 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="167e392727d993898c53993e6accaefa424605353b71ce682535fb26986182d3" Nov 02 18:54:04.524182 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:54:04.524064 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:54:08.316181 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:08.316143 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg"] Nov 02 18:54:08.316640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:08.316585 5601 cpu_manager.go:414] "containerMap: removing container" podUID="9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6" containerName="revoke-keys" Nov 02 18:54:08.316640 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:08.316599 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6" containerName="revoke-keys" Nov 02 18:54:08.316739 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:08.316703 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6" containerName="revoke-keys" Nov 02 18:54:08.319765 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:08.319750 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg" Nov 02 18:54:08.326898 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:08.326289 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:54:08.326898 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:08.326334 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:54:08.329385 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:08.328396 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg"] Nov 02 18:54:08.410579 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:08.410546 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6e66f0f-zcsqg\" (UID: \"bf94ee92-6bf4-449f-b97c-80ee9cb0c41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg" Nov 02 18:54:08.410784 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:08.410590 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82fg4\" (UniqueName: \"kubernetes.io/projected/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-kube-api-access-82fg4\") pod \"maas-api-revoke-sub-d6e66f0f-zcsqg\" (UID: \"bf94ee92-6bf4-449f-b97c-80ee9cb0c41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg" Nov 02 18:54:08.410784 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:08.410758 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6e66f0f-zcsqg\" (UID: \"bf94ee92-6bf4-449f-b97c-80ee9cb0c41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg" Nov 02 18:54:08.511344 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:08.511303 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6e66f0f-zcsqg\" (UID: \"bf94ee92-6bf4-449f-b97c-80ee9cb0c41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg" Nov 02 18:54:08.511554 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:08.511385 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6e66f0f-zcsqg\" (UID: \"bf94ee92-6bf4-449f-b97c-80ee9cb0c41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg" Nov 02 18:54:08.511554 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:08.511412 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82fg4\" (UniqueName: \"kubernetes.io/projected/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-kube-api-access-82fg4\") pod \"maas-api-revoke-sub-d6e66f0f-zcsqg\" (UID: \"bf94ee92-6bf4-449f-b97c-80ee9cb0c41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg" Nov 02 18:54:08.512202 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:08.512174 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6e66f0f-zcsqg\" (UID: \"bf94ee92-6bf4-449f-b97c-80ee9cb0c41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg" Nov 02 18:54:08.513859 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:08.513838 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6e66f0f-zcsqg\" (UID: \"bf94ee92-6bf4-449f-b97c-80ee9cb0c41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg" Nov 02 18:54:08.519238 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:08.519212 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82fg4\" (UniqueName: \"kubernetes.io/projected/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-kube-api-access-82fg4\") pod \"maas-api-revoke-sub-d6e66f0f-zcsqg\" (UID: \"bf94ee92-6bf4-449f-b97c-80ee9cb0c41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg" Nov 02 18:54:08.635710 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:08.635595 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg" Nov 02 18:54:08.907285 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:08.907219 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg"] Nov 02 18:54:08.910274 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:54:08.910245 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf94ee92_6bf4_449f_b97c_80ee9cb0c41b.slice/crio-2fc2289f64071064eb14956ee67014c4790cf03091e1f90e064c850725c3bc81 WatchSource:0}: Error finding container 2fc2289f64071064eb14956ee67014c4790cf03091e1f90e064c850725c3bc81: Status 404 returned error can't find the container with id 2fc2289f64071064eb14956ee67014c4790cf03091e1f90e064c850725c3bc81 Nov 02 18:54:09.461271 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:09.461234 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg" event={"ID":"bf94ee92-6bf4-449f-b97c-80ee9cb0c41b","Type":"ContainerStarted","Data":"e6c12c2e78e1b06d89f770c7f150505ff86a46d9b13317be3ad7d6c68af5ae1e"} Nov 02 18:54:09.461748 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:09.461297 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg" event={"ID":"bf94ee92-6bf4-449f-b97c-80ee9cb0c41b","Type":"ContainerStarted","Data":"2fc2289f64071064eb14956ee67014c4790cf03091e1f90e064c850725c3bc81"} Nov 02 18:54:09.477116 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:09.477062 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg" podStartSLOduration=1.477049173 podStartE2EDuration="1.477049173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:54:09.476671631 +0000 UTC m=+4814.167725655" watchObservedRunningTime="2026-11-02 18:54:09.477049173 +0000 UTC m=+4814.168103185" Nov 02 18:54:10.467878 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:10.467843 5601 generic.go:356] "Generic (PLEG): container finished" podID="bf94ee92-6bf4-449f-b97c-80ee9cb0c41b" containerID="e6c12c2e78e1b06d89f770c7f150505ff86a46d9b13317be3ad7d6c68af5ae1e" exitCode=0 Nov 02 18:54:10.468309 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:10.467933 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg" event={"ID":"bf94ee92-6bf4-449f-b97c-80ee9cb0c41b","Type":"ContainerDied","Data":"e6c12c2e78e1b06d89f770c7f150505ff86a46d9b13317be3ad7d6c68af5ae1e"} Nov 02 18:54:11.605270 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:11.605244 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg" Nov 02 18:54:11.742290 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:11.742211 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-maas-api-xks-ca\") pod \"bf94ee92-6bf4-449f-b97c-80ee9cb0c41b\" (UID: \"bf94ee92-6bf4-449f-b97c-80ee9cb0c41b\") " Nov 02 18:54:11.742467 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:11.742308 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-kube-api-access-82fg4\" (UniqueName: \"kubernetes.io/projected/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-kube-api-access-82fg4\") pod \"bf94ee92-6bf4-449f-b97c-80ee9cb0c41b\" (UID: \"bf94ee92-6bf4-449f-b97c-80ee9cb0c41b\") " Nov 02 18:54:11.742467 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:11.742427 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-maas-api-openshift-ca\") pod \"bf94ee92-6bf4-449f-b97c-80ee9cb0c41b\" (UID: \"bf94ee92-6bf4-449f-b97c-80ee9cb0c41b\") " Nov 02 18:54:11.742874 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:11.742845 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-maas-api-openshift-ca" pod "bf94ee92-6bf4-449f-b97c-80ee9cb0c41b" (UID: "bf94ee92-6bf4-449f-b97c-80ee9cb0c41b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:54:11.744246 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:11.744212 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-maas-api-xks-ca" pod "bf94ee92-6bf4-449f-b97c-80ee9cb0c41b" (UID: "bf94ee92-6bf4-449f-b97c-80ee9cb0c41b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:54:11.744353 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:11.744293 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-kube-api-access-82fg4" pod "bf94ee92-6bf4-449f-b97c-80ee9cb0c41b" (UID: "bf94ee92-6bf4-449f-b97c-80ee9cb0c41b"). InnerVolumeSpecName "kube-api-access-82fg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:54:11.843406 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:11.843372 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:54:11.843406 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:11.843405 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:54:11.843580 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:11.843420 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82fg4\" (UniqueName: \"kubernetes.io/projected/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b-kube-api-access-82fg4\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:54:12.479872 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:12.479838 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg" Nov 02 18:54:12.480061 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:12.479864 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg" event={"ID":"bf94ee92-6bf4-449f-b97c-80ee9cb0c41b","Type":"ContainerDied","Data":"2fc2289f64071064eb14956ee67014c4790cf03091e1f90e064c850725c3bc81"} Nov 02 18:54:12.480061 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:12.479920 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fc2289f64071064eb14956ee67014c4790cf03091e1f90e064c850725c3bc81" Nov 02 18:54:34.483748 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:54:34.483709 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:54:35.771780 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:35.771742 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495c507e-qzrt9"] Nov 02 18:54:35.772351 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:35.772279 5601 cpu_manager.go:414] "containerMap: removing container" podUID="bf94ee92-6bf4-449f-b97c-80ee9cb0c41b" containerName="revoke-keys" Nov 02 18:54:35.772351 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:35.772297 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf94ee92-6bf4-449f-b97c-80ee9cb0c41b" containerName="revoke-keys" Nov 02 18:54:35.772452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:35.772387 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf94ee92-6bf4-449f-b97c-80ee9cb0c41b" containerName="revoke-keys" Nov 02 18:54:35.775793 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:35.775776 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c507e-qzrt9" Nov 02 18:54:35.784144 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:35.784115 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:54:35.785182 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:35.784825 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495c507e-qzrt9"] Nov 02 18:54:35.785507 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:35.785486 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:54:35.915482 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:35.915441 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxc2l\" (UniqueName: \"kubernetes.io/projected/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-kube-api-access-rxc2l\") pod \"maas-api-revoke-sub-495c507e-qzrt9\" (UID: \"b5c49fcb-f23e-4366-b5eb-a0e26899e33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c507e-qzrt9" Nov 02 18:54:35.915695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:35.915500 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-495c507e-qzrt9\" (UID: \"b5c49fcb-f23e-4366-b5eb-a0e26899e33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c507e-qzrt9" Nov 02 18:54:35.915695 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:35.915640 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-495c507e-qzrt9\" (UID: \"b5c49fcb-f23e-4366-b5eb-a0e26899e33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c507e-qzrt9" Nov 02 18:54:36.017140 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:36.017105 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-495c507e-qzrt9\" (UID: \"b5c49fcb-f23e-4366-b5eb-a0e26899e33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c507e-qzrt9" Nov 02 18:54:36.017329 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:36.017170 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxc2l\" (UniqueName: \"kubernetes.io/projected/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-kube-api-access-rxc2l\") pod \"maas-api-revoke-sub-495c507e-qzrt9\" (UID: \"b5c49fcb-f23e-4366-b5eb-a0e26899e33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c507e-qzrt9" Nov 02 18:54:36.017329 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:36.017201 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-495c507e-qzrt9\" (UID: \"b5c49fcb-f23e-4366-b5eb-a0e26899e33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c507e-qzrt9" Nov 02 18:54:36.017800 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:36.017779 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-495c507e-qzrt9\" (UID: \"b5c49fcb-f23e-4366-b5eb-a0e26899e33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c507e-qzrt9" Nov 02 18:54:36.019526 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:36.019504 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-495c507e-qzrt9\" (UID: \"b5c49fcb-f23e-4366-b5eb-a0e26899e33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c507e-qzrt9" Nov 02 18:54:36.025095 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:36.025043 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxc2l\" (UniqueName: \"kubernetes.io/projected/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-kube-api-access-rxc2l\") pod \"maas-api-revoke-sub-495c507e-qzrt9\" (UID: \"b5c49fcb-f23e-4366-b5eb-a0e26899e33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c507e-qzrt9" Nov 02 18:54:36.092755 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:36.092709 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c507e-qzrt9" Nov 02 18:54:36.363024 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:36.362991 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495c507e-qzrt9"] Nov 02 18:54:36.365355 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:54:36.365328 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c49fcb_f23e_4366_b5eb_a0e26899e33d.slice/crio-475ae8f4a7a8af8eadcb8867676e4f99d40f900e3a4b5370399cd8aeb1a7bbac WatchSource:0}: Error finding container 475ae8f4a7a8af8eadcb8867676e4f99d40f900e3a4b5370399cd8aeb1a7bbac: Status 404 returned error can't find the container with id 475ae8f4a7a8af8eadcb8867676e4f99d40f900e3a4b5370399cd8aeb1a7bbac Nov 02 18:54:36.628336 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:36.628248 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c507e-qzrt9" event={"ID":"b5c49fcb-f23e-4366-b5eb-a0e26899e33d","Type":"ContainerStarted","Data":"475ae8f4a7a8af8eadcb8867676e4f99d40f900e3a4b5370399cd8aeb1a7bbac"} Nov 02 18:54:37.635107 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:37.635071 5601 generic.go:356] "Generic (PLEG): container finished" podID="b5c49fcb-f23e-4366-b5eb-a0e26899e33d" containerID="c4c3a5b94aaee4adca97fe2c5f369f70e90d1309b04956d96f1dcc835f1b77c0" exitCode=0 Nov 02 18:54:37.635518 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:37.635156 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c507e-qzrt9" event={"ID":"b5c49fcb-f23e-4366-b5eb-a0e26899e33d","Type":"ContainerDied","Data":"c4c3a5b94aaee4adca97fe2c5f369f70e90d1309b04956d96f1dcc835f1b77c0"} Nov 02 18:54:38.777859 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:38.777831 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c507e-qzrt9" Nov 02 18:54:38.844796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:38.844760 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-kube-api-access-rxc2l\" (UniqueName: \"kubernetes.io/projected/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-kube-api-access-rxc2l\") pod \"b5c49fcb-f23e-4366-b5eb-a0e26899e33d\" (UID: \"b5c49fcb-f23e-4366-b5eb-a0e26899e33d\") " Nov 02 18:54:38.844796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:38.844801 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-maas-api-xks-ca\") pod \"b5c49fcb-f23e-4366-b5eb-a0e26899e33d\" (UID: \"b5c49fcb-f23e-4366-b5eb-a0e26899e33d\") " Nov 02 18:54:38.845017 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:38.844838 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-maas-api-openshift-ca\") pod \"b5c49fcb-f23e-4366-b5eb-a0e26899e33d\" (UID: \"b5c49fcb-f23e-4366-b5eb-a0e26899e33d\") " Nov 02 18:54:38.845271 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:38.845248 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-maas-api-openshift-ca" pod "b5c49fcb-f23e-4366-b5eb-a0e26899e33d" (UID: "b5c49fcb-f23e-4366-b5eb-a0e26899e33d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:54:38.847016 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:38.846986 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-kube-api-access-rxc2l" pod "b5c49fcb-f23e-4366-b5eb-a0e26899e33d" (UID: "b5c49fcb-f23e-4366-b5eb-a0e26899e33d"). InnerVolumeSpecName "kube-api-access-rxc2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:54:38.847141 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:38.847104 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-maas-api-xks-ca" pod "b5c49fcb-f23e-4366-b5eb-a0e26899e33d" (UID: "b5c49fcb-f23e-4366-b5eb-a0e26899e33d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:54:38.946299 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:38.946260 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:54:38.946299 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:38.946295 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxc2l\" (UniqueName: \"kubernetes.io/projected/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-kube-api-access-rxc2l\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:54:38.946299 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:38.946306 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5c49fcb-f23e-4366-b5eb-a0e26899e33d-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:54:39.649889 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:39.649853 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c507e-qzrt9" event={"ID":"b5c49fcb-f23e-4366-b5eb-a0e26899e33d","Type":"ContainerDied","Data":"475ae8f4a7a8af8eadcb8867676e4f99d40f900e3a4b5370399cd8aeb1a7bbac"} Nov 02 18:54:39.650083 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:39.649909 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="475ae8f4a7a8af8eadcb8867676e4f99d40f900e3a4b5370399cd8aeb1a7bbac" Nov 02 18:54:39.650083 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:39.649872 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c507e-qzrt9" Nov 02 18:54:46.037834 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.037794 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ab0ecc-462bv"] Nov 02 18:54:46.039350 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.039319 5601 cpu_manager.go:414] "containerMap: removing container" podUID="b5c49fcb-f23e-4366-b5eb-a0e26899e33d" containerName="revoke-keys" Nov 02 18:54:46.039527 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.039512 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c49fcb-f23e-4366-b5eb-a0e26899e33d" containerName="revoke-keys" Nov 02 18:54:46.039843 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.039825 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c49fcb-f23e-4366-b5eb-a0e26899e33d" containerName="revoke-keys" Nov 02 18:54:46.044125 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.044100 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ab0ecc-462bv" Nov 02 18:54:46.049826 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.049789 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:54:46.049975 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.049943 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:54:46.051440 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.051411 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ab0ecc-462bv"] Nov 02 18:54:46.221088 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.221044 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq7bb\" (UniqueName: \"kubernetes.io/projected/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-kube-api-access-kq7bb\") pod \"maas-api-revoke-sub-97ab0ecc-462bv\" (UID: \"0a3483e4-4266-4bfd-a01a-a6cb0aed3089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ab0ecc-462bv" Nov 02 18:54:46.221280 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.221180 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97ab0ecc-462bv\" (UID: \"0a3483e4-4266-4bfd-a01a-a6cb0aed3089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ab0ecc-462bv" Nov 02 18:54:46.221280 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.221250 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97ab0ecc-462bv\" (UID: \"0a3483e4-4266-4bfd-a01a-a6cb0aed3089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ab0ecc-462bv" Nov 02 18:54:46.322664 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.322518 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97ab0ecc-462bv\" (UID: \"0a3483e4-4266-4bfd-a01a-a6cb0aed3089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ab0ecc-462bv" Nov 02 18:54:46.322664 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.322588 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97ab0ecc-462bv\" (UID: \"0a3483e4-4266-4bfd-a01a-a6cb0aed3089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ab0ecc-462bv" Nov 02 18:54:46.322664 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.322657 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq7bb\" (UniqueName: \"kubernetes.io/projected/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-kube-api-access-kq7bb\") pod \"maas-api-revoke-sub-97ab0ecc-462bv\" (UID: \"0a3483e4-4266-4bfd-a01a-a6cb0aed3089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ab0ecc-462bv" Nov 02 18:54:46.323321 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.323291 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97ab0ecc-462bv\" (UID: \"0a3483e4-4266-4bfd-a01a-a6cb0aed3089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ab0ecc-462bv" Nov 02 18:54:46.325078 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.325053 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97ab0ecc-462bv\" (UID: \"0a3483e4-4266-4bfd-a01a-a6cb0aed3089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ab0ecc-462bv" Nov 02 18:54:46.331201 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.331172 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq7bb\" (UniqueName: \"kubernetes.io/projected/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-kube-api-access-kq7bb\") pod \"maas-api-revoke-sub-97ab0ecc-462bv\" (UID: \"0a3483e4-4266-4bfd-a01a-a6cb0aed3089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ab0ecc-462bv" Nov 02 18:54:46.360936 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.360893 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ab0ecc-462bv" Nov 02 18:54:46.635111 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.635080 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ab0ecc-462bv"] Nov 02 18:54:46.637678 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:54:46.637610 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3483e4_4266_4bfd_a01a_a6cb0aed3089.slice/crio-945e8c14d9be3940a907cceb56386c6abbd7cef6350886dadb81fdaec42e24e0 WatchSource:0}: Error finding container 945e8c14d9be3940a907cceb56386c6abbd7cef6350886dadb81fdaec42e24e0: Status 404 returned error can't find the container with id 945e8c14d9be3940a907cceb56386c6abbd7cef6350886dadb81fdaec42e24e0 Nov 02 18:54:46.685868 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:46.685835 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ab0ecc-462bv" event={"ID":"0a3483e4-4266-4bfd-a01a-a6cb0aed3089","Type":"ContainerStarted","Data":"945e8c14d9be3940a907cceb56386c6abbd7cef6350886dadb81fdaec42e24e0"} Nov 02 18:54:47.692186 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:47.692138 5601 generic.go:356] "Generic (PLEG): container finished" podID="0a3483e4-4266-4bfd-a01a-a6cb0aed3089" containerID="baf94653dc0cd95ceacfecad980708a16bc3ffe56bb4751e1355953fb040b358" exitCode=0 Nov 02 18:54:47.692602 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:47.692199 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ab0ecc-462bv" event={"ID":"0a3483e4-4266-4bfd-a01a-a6cb0aed3089","Type":"ContainerDied","Data":"baf94653dc0cd95ceacfecad980708a16bc3ffe56bb4751e1355953fb040b358"} Nov 02 18:54:48.825406 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:48.825377 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ab0ecc-462bv" Nov 02 18:54:48.949995 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:48.949909 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-maas-api-xks-ca\") pod \"0a3483e4-4266-4bfd-a01a-a6cb0aed3089\" (UID: \"0a3483e4-4266-4bfd-a01a-a6cb0aed3089\") " Nov 02 18:54:48.949995 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:48.949987 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-kube-api-access-kq7bb\" (UniqueName: \"kubernetes.io/projected/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-kube-api-access-kq7bb\") pod \"0a3483e4-4266-4bfd-a01a-a6cb0aed3089\" (UID: \"0a3483e4-4266-4bfd-a01a-a6cb0aed3089\") " Nov 02 18:54:48.950249 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:48.950018 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-maas-api-openshift-ca\") pod \"0a3483e4-4266-4bfd-a01a-a6cb0aed3089\" (UID: \"0a3483e4-4266-4bfd-a01a-a6cb0aed3089\") " Nov 02 18:54:48.950448 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:48.950423 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-maas-api-openshift-ca" pod "0a3483e4-4266-4bfd-a01a-a6cb0aed3089" (UID: "0a3483e4-4266-4bfd-a01a-a6cb0aed3089"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:54:48.952110 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:48.952089 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-maas-api-xks-ca" pod "0a3483e4-4266-4bfd-a01a-a6cb0aed3089" (UID: "0a3483e4-4266-4bfd-a01a-a6cb0aed3089"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:54:48.952198 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:48.952164 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-kube-api-access-kq7bb" pod "0a3483e4-4266-4bfd-a01a-a6cb0aed3089" (UID: "0a3483e4-4266-4bfd-a01a-a6cb0aed3089"). InnerVolumeSpecName "kube-api-access-kq7bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:54:49.051448 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:49.051409 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:54:49.051448 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:49.051441 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:54:49.051448 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:49.051455 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq7bb\" (UniqueName: \"kubernetes.io/projected/0a3483e4-4266-4bfd-a01a-a6cb0aed3089-kube-api-access-kq7bb\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:54:49.702961 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:49.702881 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ab0ecc-462bv" Nov 02 18:54:49.703113 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:49.702881 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ab0ecc-462bv" event={"ID":"0a3483e4-4266-4bfd-a01a-a6cb0aed3089","Type":"ContainerDied","Data":"945e8c14d9be3940a907cceb56386c6abbd7cef6350886dadb81fdaec42e24e0"} Nov 02 18:54:49.703113 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:54:49.703009 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="945e8c14d9be3940a907cceb56386c6abbd7cef6350886dadb81fdaec42e24e0" Nov 02 18:55:01.777309 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:01.777215 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-jv8f4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:55:01.777824 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:01.777304 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-jv8f4" podUID="c32fa6f3-9665-4b3b-b093-22435190829b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:55:04.528747 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:55:04.528704 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:55:12.702591 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:12.702551 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz"] Nov 02 18:55:12.703013 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:12.702980 5601 cpu_manager.go:414] "containerMap: removing container" podUID="0a3483e4-4266-4bfd-a01a-a6cb0aed3089" containerName="revoke-keys" Nov 02 18:55:12.703013 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:12.702992 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a3483e4-4266-4bfd-a01a-a6cb0aed3089" containerName="revoke-keys" Nov 02 18:55:12.703107 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:12.703064 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a3483e4-4266-4bfd-a01a-a6cb0aed3089" containerName="revoke-keys" Nov 02 18:55:12.706241 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:12.706222 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz" Nov 02 18:55:12.711951 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:12.711925 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:55:12.712257 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:12.712233 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:55:12.713513 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:12.713486 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz"] Nov 02 18:55:12.744709 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:12.744664 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e715639-a159-4bce-8e35-812b50f35ae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25c9f82-x7wdz\" (UID: \"4e715639-a159-4bce-8e35-812b50f35ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz" Nov 02 18:55:12.744877 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:12.744803 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e715639-a159-4bce-8e35-812b50f35ae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25c9f82-x7wdz\" (UID: \"4e715639-a159-4bce-8e35-812b50f35ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz" Nov 02 18:55:12.744877 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:12.744831 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pntp\" (UniqueName: \"kubernetes.io/projected/4e715639-a159-4bce-8e35-812b50f35ae1-kube-api-access-2pntp\") pod \"maas-api-revoke-sub-f25c9f82-x7wdz\" (UID: \"4e715639-a159-4bce-8e35-812b50f35ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz" Nov 02 18:55:12.846011 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:12.845977 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e715639-a159-4bce-8e35-812b50f35ae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25c9f82-x7wdz\" (UID: \"4e715639-a159-4bce-8e35-812b50f35ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz" Nov 02 18:55:12.846192 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:12.846042 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e715639-a159-4bce-8e35-812b50f35ae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25c9f82-x7wdz\" (UID: \"4e715639-a159-4bce-8e35-812b50f35ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz" Nov 02 18:55:12.846192 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:12.846060 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pntp\" (UniqueName: \"kubernetes.io/projected/4e715639-a159-4bce-8e35-812b50f35ae1-kube-api-access-2pntp\") pod \"maas-api-revoke-sub-f25c9f82-x7wdz\" (UID: \"4e715639-a159-4bce-8e35-812b50f35ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz" Nov 02 18:55:12.846792 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:12.846763 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e715639-a159-4bce-8e35-812b50f35ae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25c9f82-x7wdz\" (UID: \"4e715639-a159-4bce-8e35-812b50f35ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz" Nov 02 18:55:12.848400 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:12.848383 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e715639-a159-4bce-8e35-812b50f35ae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25c9f82-x7wdz\" (UID: \"4e715639-a159-4bce-8e35-812b50f35ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz" Nov 02 18:55:12.854931 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:12.854896 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pntp\" (UniqueName: \"kubernetes.io/projected/4e715639-a159-4bce-8e35-812b50f35ae1-kube-api-access-2pntp\") pod \"maas-api-revoke-sub-f25c9f82-x7wdz\" (UID: \"4e715639-a159-4bce-8e35-812b50f35ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz" Nov 02 18:55:13.021893 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:13.021807 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz" Nov 02 18:55:13.300913 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:13.300885 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz"] Nov 02 18:55:13.302819 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:55:13.302790 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e715639_a159_4bce_8e35_812b50f35ae1.slice/crio-dbd7bdf387f15bb1ca05827eba25b065a19590c0bd4e7a0ada95887d9a6f3b0a WatchSource:0}: Error finding container dbd7bdf387f15bb1ca05827eba25b065a19590c0bd4e7a0ada95887d9a6f3b0a: Status 404 returned error can't find the container with id dbd7bdf387f15bb1ca05827eba25b065a19590c0bd4e7a0ada95887d9a6f3b0a Nov 02 18:55:13.851250 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:13.851211 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz" event={"ID":"4e715639-a159-4bce-8e35-812b50f35ae1","Type":"ContainerStarted","Data":"c5fba70481d394a25bbae1ca0a6ea9596d88ed338002aadee947a25108e87384"} Nov 02 18:55:13.851782 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:13.851262 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz" event={"ID":"4e715639-a159-4bce-8e35-812b50f35ae1","Type":"ContainerStarted","Data":"dbd7bdf387f15bb1ca05827eba25b065a19590c0bd4e7a0ada95887d9a6f3b0a"} Nov 02 18:55:13.868054 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:13.867996 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz" podStartSLOduration=1.867974992 podStartE2EDuration="1.867974992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:55:13.86738826 +0000 UTC m=+4878.558442266" watchObservedRunningTime="2026-11-02 18:55:13.867974992 +0000 UTC m=+4878.559029005" Nov 02 18:55:14.857701 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:14.857663 5601 generic.go:356] "Generic (PLEG): container finished" podID="4e715639-a159-4bce-8e35-812b50f35ae1" containerID="c5fba70481d394a25bbae1ca0a6ea9596d88ed338002aadee947a25108e87384" exitCode=0 Nov 02 18:55:14.858131 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:14.857751 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz" event={"ID":"4e715639-a159-4bce-8e35-812b50f35ae1","Type":"ContainerDied","Data":"c5fba70481d394a25bbae1ca0a6ea9596d88ed338002aadee947a25108e87384"} Nov 02 18:55:16.004801 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:16.004769 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz" Nov 02 18:55:16.077682 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:16.077643 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e715639-a159-4bce-8e35-812b50f35ae1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e715639-a159-4bce-8e35-812b50f35ae1-maas-api-openshift-ca\") pod \"4e715639-a159-4bce-8e35-812b50f35ae1\" (UID: \"4e715639-a159-4bce-8e35-812b50f35ae1\") " Nov 02 18:55:16.077877 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:16.077730 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e715639-a159-4bce-8e35-812b50f35ae1-kube-api-access-2pntp\" (UniqueName: \"kubernetes.io/projected/4e715639-a159-4bce-8e35-812b50f35ae1-kube-api-access-2pntp\") pod \"4e715639-a159-4bce-8e35-812b50f35ae1\" (UID: \"4e715639-a159-4bce-8e35-812b50f35ae1\") " Nov 02 18:55:16.077877 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:16.077762 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e715639-a159-4bce-8e35-812b50f35ae1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e715639-a159-4bce-8e35-812b50f35ae1-maas-api-xks-ca\") pod \"4e715639-a159-4bce-8e35-812b50f35ae1\" (UID: \"4e715639-a159-4bce-8e35-812b50f35ae1\") " Nov 02 18:55:16.078033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:16.078007 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e715639-a159-4bce-8e35-812b50f35ae1-maas-api-openshift-ca" pod "4e715639-a159-4bce-8e35-812b50f35ae1" (UID: "4e715639-a159-4bce-8e35-812b50f35ae1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:55:16.079981 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:16.079943 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e715639-a159-4bce-8e35-812b50f35ae1-maas-api-xks-ca" pod "4e715639-a159-4bce-8e35-812b50f35ae1" (UID: "4e715639-a159-4bce-8e35-812b50f35ae1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:55:16.080113 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:16.079993 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e715639-a159-4bce-8e35-812b50f35ae1-kube-api-access-2pntp" pod "4e715639-a159-4bce-8e35-812b50f35ae1" (UID: "4e715639-a159-4bce-8e35-812b50f35ae1"). InnerVolumeSpecName "kube-api-access-2pntp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:55:16.179152 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:16.179116 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e715639-a159-4bce-8e35-812b50f35ae1-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:55:16.179152 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:16.179149 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pntp\" (UniqueName: \"kubernetes.io/projected/4e715639-a159-4bce-8e35-812b50f35ae1-kube-api-access-2pntp\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:55:16.179362 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:16.179160 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e715639-a159-4bce-8e35-812b50f35ae1-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:55:16.870331 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:16.870287 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz" event={"ID":"4e715639-a159-4bce-8e35-812b50f35ae1","Type":"ContainerDied","Data":"dbd7bdf387f15bb1ca05827eba25b065a19590c0bd4e7a0ada95887d9a6f3b0a"} Nov 02 18:55:16.870331 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:16.870328 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25c9f82-x7wdz" Nov 02 18:55:16.870556 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:16.870389 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbd7bdf387f15bb1ca05827eba25b065a19590c0bd4e7a0ada95887d9a6f3b0a" Nov 02 18:55:22.923168 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:22.923134 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt"] Nov 02 18:55:22.923683 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:22.923547 5601 cpu_manager.go:414] "containerMap: removing container" podUID="4e715639-a159-4bce-8e35-812b50f35ae1" containerName="revoke-keys" Nov 02 18:55:22.923683 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:22.923560 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e715639-a159-4bce-8e35-812b50f35ae1" containerName="revoke-keys" Nov 02 18:55:22.923683 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:22.923631 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e715639-a159-4bce-8e35-812b50f35ae1" containerName="revoke-keys" Nov 02 18:55:22.926905 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:22.926879 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt" Nov 02 18:55:22.935700 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:22.934596 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:55:22.935700 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:22.934788 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:55:22.938892 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:22.937946 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt"] Nov 02 18:55:23.044807 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:23.044768 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750a8b45-b55f-45fd-8f3b-1858245a4e08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e73b344f-vz6dt\" (UID: \"750a8b45-b55f-45fd-8f3b-1858245a4e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt" Nov 02 18:55:23.044990 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:23.044841 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750a8b45-b55f-45fd-8f3b-1858245a4e08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e73b344f-vz6dt\" (UID: \"750a8b45-b55f-45fd-8f3b-1858245a4e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt" Nov 02 18:55:23.044990 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:23.044967 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ww6f\" (UniqueName: \"kubernetes.io/projected/750a8b45-b55f-45fd-8f3b-1858245a4e08-kube-api-access-8ww6f\") pod \"maas-api-revoke-sub-e73b344f-vz6dt\" (UID: \"750a8b45-b55f-45fd-8f3b-1858245a4e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt" Nov 02 18:55:23.145642 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:23.145583 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750a8b45-b55f-45fd-8f3b-1858245a4e08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e73b344f-vz6dt\" (UID: \"750a8b45-b55f-45fd-8f3b-1858245a4e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt" Nov 02 18:55:23.145831 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:23.145709 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ww6f\" (UniqueName: \"kubernetes.io/projected/750a8b45-b55f-45fd-8f3b-1858245a4e08-kube-api-access-8ww6f\") pod \"maas-api-revoke-sub-e73b344f-vz6dt\" (UID: \"750a8b45-b55f-45fd-8f3b-1858245a4e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt" Nov 02 18:55:23.145831 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:23.145761 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750a8b45-b55f-45fd-8f3b-1858245a4e08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e73b344f-vz6dt\" (UID: \"750a8b45-b55f-45fd-8f3b-1858245a4e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt" Nov 02 18:55:23.146389 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:23.146366 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750a8b45-b55f-45fd-8f3b-1858245a4e08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e73b344f-vz6dt\" (UID: \"750a8b45-b55f-45fd-8f3b-1858245a4e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt" Nov 02 18:55:23.148029 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:23.148008 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750a8b45-b55f-45fd-8f3b-1858245a4e08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e73b344f-vz6dt\" (UID: \"750a8b45-b55f-45fd-8f3b-1858245a4e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt" Nov 02 18:55:23.153489 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:23.153467 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ww6f\" (UniqueName: \"kubernetes.io/projected/750a8b45-b55f-45fd-8f3b-1858245a4e08-kube-api-access-8ww6f\") pod \"maas-api-revoke-sub-e73b344f-vz6dt\" (UID: \"750a8b45-b55f-45fd-8f3b-1858245a4e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt" Nov 02 18:55:23.250164 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:23.250090 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt" Nov 02 18:55:23.524524 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:23.524483 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt"] Nov 02 18:55:23.527722 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:55:23.527693 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750a8b45_b55f_45fd_8f3b_1858245a4e08.slice/crio-e5ab18f79755aa38ec869d6b8f2e1c38ff0281c83faa7e1e47b7e8919fc36f34 WatchSource:0}: Error finding container e5ab18f79755aa38ec869d6b8f2e1c38ff0281c83faa7e1e47b7e8919fc36f34: Status 404 returned error can't find the container with id e5ab18f79755aa38ec869d6b8f2e1c38ff0281c83faa7e1e47b7e8919fc36f34 Nov 02 18:55:23.908778 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:23.908737 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt" event={"ID":"750a8b45-b55f-45fd-8f3b-1858245a4e08","Type":"ContainerStarted","Data":"9bbfbc5607f18a48e2f54750912959f5ed68d24f0fcd2e9e0261187b07b1765c"} Nov 02 18:55:23.908917 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:23.908803 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt" event={"ID":"750a8b45-b55f-45fd-8f3b-1858245a4e08","Type":"ContainerStarted","Data":"e5ab18f79755aa38ec869d6b8f2e1c38ff0281c83faa7e1e47b7e8919fc36f34"} Nov 02 18:55:23.943489 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:23.943428 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt" podStartSLOduration=1.943412237 podStartE2EDuration="1.943412237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:55:23.940541592 +0000 UTC m=+4888.631595603" watchObservedRunningTime="2026-11-02 18:55:23.943412237 +0000 UTC m=+4888.634466252" Nov 02 18:55:24.920086 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:24.920050 5601 generic.go:356] "Generic (PLEG): container finished" podID="750a8b45-b55f-45fd-8f3b-1858245a4e08" containerID="9bbfbc5607f18a48e2f54750912959f5ed68d24f0fcd2e9e0261187b07b1765c" exitCode=0 Nov 02 18:55:24.920304 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:24.920138 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt" event={"ID":"750a8b45-b55f-45fd-8f3b-1858245a4e08","Type":"ContainerDied","Data":"9bbfbc5607f18a48e2f54750912959f5ed68d24f0fcd2e9e0261187b07b1765c"} Nov 02 18:55:26.317211 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:26.317184 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt" Nov 02 18:55:26.417311 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:26.417272 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/750a8b45-b55f-45fd-8f3b-1858245a4e08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750a8b45-b55f-45fd-8f3b-1858245a4e08-maas-api-xks-ca\") pod \"750a8b45-b55f-45fd-8f3b-1858245a4e08\" (UID: \"750a8b45-b55f-45fd-8f3b-1858245a4e08\") " Nov 02 18:55:26.417509 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:26.417329 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/750a8b45-b55f-45fd-8f3b-1858245a4e08-kube-api-access-8ww6f\" (UniqueName: \"kubernetes.io/projected/750a8b45-b55f-45fd-8f3b-1858245a4e08-kube-api-access-8ww6f\") pod \"750a8b45-b55f-45fd-8f3b-1858245a4e08\" (UID: \"750a8b45-b55f-45fd-8f3b-1858245a4e08\") " Nov 02 18:55:26.417509 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:26.417364 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/750a8b45-b55f-45fd-8f3b-1858245a4e08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750a8b45-b55f-45fd-8f3b-1858245a4e08-maas-api-openshift-ca\") pod \"750a8b45-b55f-45fd-8f3b-1858245a4e08\" (UID: \"750a8b45-b55f-45fd-8f3b-1858245a4e08\") " Nov 02 18:55:26.418040 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:26.418003 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750a8b45-b55f-45fd-8f3b-1858245a4e08-maas-api-openshift-ca" pod "750a8b45-b55f-45fd-8f3b-1858245a4e08" (UID: "750a8b45-b55f-45fd-8f3b-1858245a4e08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:55:26.419864 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:26.419834 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/750a8b45-b55f-45fd-8f3b-1858245a4e08-maas-api-xks-ca" pod "750a8b45-b55f-45fd-8f3b-1858245a4e08" (UID: "750a8b45-b55f-45fd-8f3b-1858245a4e08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:55:26.419987 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:26.419962 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750a8b45-b55f-45fd-8f3b-1858245a4e08-kube-api-access-8ww6f" pod "750a8b45-b55f-45fd-8f3b-1858245a4e08" (UID: "750a8b45-b55f-45fd-8f3b-1858245a4e08"). InnerVolumeSpecName "kube-api-access-8ww6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:55:26.518380 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:26.518340 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750a8b45-b55f-45fd-8f3b-1858245a4e08-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:55:26.518656 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:26.518607 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ww6f\" (UniqueName: \"kubernetes.io/projected/750a8b45-b55f-45fd-8f3b-1858245a4e08-kube-api-access-8ww6f\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:55:26.518774 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:26.518659 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750a8b45-b55f-45fd-8f3b-1858245a4e08-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:55:26.930988 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:26.930952 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt" event={"ID":"750a8b45-b55f-45fd-8f3b-1858245a4e08","Type":"ContainerDied","Data":"e5ab18f79755aa38ec869d6b8f2e1c38ff0281c83faa7e1e47b7e8919fc36f34"} Nov 02 18:55:26.931174 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:26.931002 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5ab18f79755aa38ec869d6b8f2e1c38ff0281c83faa7e1e47b7e8919fc36f34" Nov 02 18:55:26.931174 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:26.930972 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b344f-vz6dt" Nov 02 18:55:34.484450 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:55:34.484406 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:55:53.327138 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:53.327100 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20fbe5c3-dn5vn"] Nov 02 18:55:53.327566 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:53.327472 5601 cpu_manager.go:414] "containerMap: removing container" podUID="750a8b45-b55f-45fd-8f3b-1858245a4e08" containerName="revoke-keys" Nov 02 18:55:53.327566 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:53.327482 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750a8b45-b55f-45fd-8f3b-1858245a4e08" containerName="revoke-keys" Nov 02 18:55:53.327566 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:53.327565 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="750a8b45-b55f-45fd-8f3b-1858245a4e08" containerName="revoke-keys" Nov 02 18:55:53.330827 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:53.330810 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fbe5c3-dn5vn" Nov 02 18:55:53.336468 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:53.336441 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:55:53.336637 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:53.336479 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:55:53.338149 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:53.338124 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20fbe5c3-dn5vn"] Nov 02 18:55:53.480949 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:53.480906 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c23979e-84bc-45f9-a989-13d2989bd8d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20fbe5c3-dn5vn\" (UID: \"4c23979e-84bc-45f9-a989-13d2989bd8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fbe5c3-dn5vn" Nov 02 18:55:53.481134 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:53.480968 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c23979e-84bc-45f9-a989-13d2989bd8d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20fbe5c3-dn5vn\" (UID: \"4c23979e-84bc-45f9-a989-13d2989bd8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fbe5c3-dn5vn" Nov 02 18:55:53.481180 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:53.481130 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l7n5\" (UniqueName: \"kubernetes.io/projected/4c23979e-84bc-45f9-a989-13d2989bd8d6-kube-api-access-9l7n5\") pod \"maas-api-revoke-sub-20fbe5c3-dn5vn\" (UID: \"4c23979e-84bc-45f9-a989-13d2989bd8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fbe5c3-dn5vn" Nov 02 18:55:53.582443 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:53.582358 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l7n5\" (UniqueName: \"kubernetes.io/projected/4c23979e-84bc-45f9-a989-13d2989bd8d6-kube-api-access-9l7n5\") pod \"maas-api-revoke-sub-20fbe5c3-dn5vn\" (UID: \"4c23979e-84bc-45f9-a989-13d2989bd8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fbe5c3-dn5vn" Nov 02 18:55:53.582443 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:53.582400 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c23979e-84bc-45f9-a989-13d2989bd8d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20fbe5c3-dn5vn\" (UID: \"4c23979e-84bc-45f9-a989-13d2989bd8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fbe5c3-dn5vn" Nov 02 18:55:53.582443 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:53.582430 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c23979e-84bc-45f9-a989-13d2989bd8d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20fbe5c3-dn5vn\" (UID: \"4c23979e-84bc-45f9-a989-13d2989bd8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fbe5c3-dn5vn" Nov 02 18:55:53.583088 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:53.583065 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c23979e-84bc-45f9-a989-13d2989bd8d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20fbe5c3-dn5vn\" (UID: \"4c23979e-84bc-45f9-a989-13d2989bd8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fbe5c3-dn5vn" Nov 02 18:55:53.584826 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:53.584804 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c23979e-84bc-45f9-a989-13d2989bd8d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20fbe5c3-dn5vn\" (UID: \"4c23979e-84bc-45f9-a989-13d2989bd8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fbe5c3-dn5vn" Nov 02 18:55:53.591209 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:53.591181 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l7n5\" (UniqueName: \"kubernetes.io/projected/4c23979e-84bc-45f9-a989-13d2989bd8d6-kube-api-access-9l7n5\") pod \"maas-api-revoke-sub-20fbe5c3-dn5vn\" (UID: \"4c23979e-84bc-45f9-a989-13d2989bd8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fbe5c3-dn5vn" Nov 02 18:55:53.646442 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:53.646417 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fbe5c3-dn5vn" Nov 02 18:55:53.922219 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:53.922190 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20fbe5c3-dn5vn"] Nov 02 18:55:53.924494 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:55:53.924466 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c23979e_84bc_45f9_a989_13d2989bd8d6.slice/crio-9bfde09f6a1dfda72439c7840f7b88bc483c72fbeba80a693937d0cc86bbf547 WatchSource:0}: Error finding container 9bfde09f6a1dfda72439c7840f7b88bc483c72fbeba80a693937d0cc86bbf547: Status 404 returned error can't find the container with id 9bfde09f6a1dfda72439c7840f7b88bc483c72fbeba80a693937d0cc86bbf547 Nov 02 18:55:54.166108 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:54.166076 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fbe5c3-dn5vn" event={"ID":"4c23979e-84bc-45f9-a989-13d2989bd8d6","Type":"ContainerStarted","Data":"9bfde09f6a1dfda72439c7840f7b88bc483c72fbeba80a693937d0cc86bbf547"} Nov 02 18:55:55.173218 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:55.173183 5601 generic.go:356] "Generic (PLEG): container finished" podID="4c23979e-84bc-45f9-a989-13d2989bd8d6" containerID="263b988f102a2e2596b2dfbbdeb1c4365fd521ea2411fb3c38706c472ac9d529" exitCode=0 Nov 02 18:55:55.173600 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:55.173268 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fbe5c3-dn5vn" event={"ID":"4c23979e-84bc-45f9-a989-13d2989bd8d6","Type":"ContainerDied","Data":"263b988f102a2e2596b2dfbbdeb1c4365fd521ea2411fb3c38706c472ac9d529"} Nov 02 18:55:56.309019 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:56.308990 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fbe5c3-dn5vn" Nov 02 18:55:56.409785 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:56.409748 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c23979e-84bc-45f9-a989-13d2989bd8d6-kube-api-access-9l7n5\" (UniqueName: \"kubernetes.io/projected/4c23979e-84bc-45f9-a989-13d2989bd8d6-kube-api-access-9l7n5\") pod \"4c23979e-84bc-45f9-a989-13d2989bd8d6\" (UID: \"4c23979e-84bc-45f9-a989-13d2989bd8d6\") " Nov 02 18:55:56.409967 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:56.409832 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c23979e-84bc-45f9-a989-13d2989bd8d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c23979e-84bc-45f9-a989-13d2989bd8d6-maas-api-openshift-ca\") pod \"4c23979e-84bc-45f9-a989-13d2989bd8d6\" (UID: \"4c23979e-84bc-45f9-a989-13d2989bd8d6\") " Nov 02 18:55:56.409967 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:56.409884 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c23979e-84bc-45f9-a989-13d2989bd8d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c23979e-84bc-45f9-a989-13d2989bd8d6-maas-api-xks-ca\") pod \"4c23979e-84bc-45f9-a989-13d2989bd8d6\" (UID: \"4c23979e-84bc-45f9-a989-13d2989bd8d6\") " Nov 02 18:55:56.410188 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:56.410156 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c23979e-84bc-45f9-a989-13d2989bd8d6-maas-api-openshift-ca" pod "4c23979e-84bc-45f9-a989-13d2989bd8d6" (UID: "4c23979e-84bc-45f9-a989-13d2989bd8d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:55:56.412018 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:56.411995 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c23979e-84bc-45f9-a989-13d2989bd8d6-kube-api-access-9l7n5" pod "4c23979e-84bc-45f9-a989-13d2989bd8d6" (UID: "4c23979e-84bc-45f9-a989-13d2989bd8d6"). InnerVolumeSpecName "kube-api-access-9l7n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:55:56.412088 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:56.412071 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c23979e-84bc-45f9-a989-13d2989bd8d6-maas-api-xks-ca" pod "4c23979e-84bc-45f9-a989-13d2989bd8d6" (UID: "4c23979e-84bc-45f9-a989-13d2989bd8d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:55:56.511036 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:56.510964 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l7n5\" (UniqueName: \"kubernetes.io/projected/4c23979e-84bc-45f9-a989-13d2989bd8d6-kube-api-access-9l7n5\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:55:56.511036 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:56.510994 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c23979e-84bc-45f9-a989-13d2989bd8d6-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:55:56.511036 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:56.511005 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c23979e-84bc-45f9-a989-13d2989bd8d6-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:55:57.185755 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:57.185721 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fbe5c3-dn5vn" Nov 02 18:55:57.185962 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:57.185753 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fbe5c3-dn5vn" event={"ID":"4c23979e-84bc-45f9-a989-13d2989bd8d6","Type":"ContainerDied","Data":"9bfde09f6a1dfda72439c7840f7b88bc483c72fbeba80a693937d0cc86bbf547"} Nov 02 18:55:57.185962 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:55:57.185803 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bfde09f6a1dfda72439c7840f7b88bc483c72fbeba80a693937d0cc86bbf547" Nov 02 18:56:03.544594 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:03.544562 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0245bb0d-xtfvr"] Nov 02 18:56:03.545006 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:03.544949 5601 cpu_manager.go:414] "containerMap: removing container" podUID="4c23979e-84bc-45f9-a989-13d2989bd8d6" containerName="revoke-keys" Nov 02 18:56:03.545006 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:03.544962 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c23979e-84bc-45f9-a989-13d2989bd8d6" containerName="revoke-keys" Nov 02 18:56:03.545090 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:03.545040 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c23979e-84bc-45f9-a989-13d2989bd8d6" containerName="revoke-keys" Nov 02 18:56:03.548377 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:03.548351 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0245bb0d-xtfvr" Nov 02 18:56:03.555961 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:03.555796 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:56:03.555961 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:03.555874 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0245bb0d-xtfvr"] Nov 02 18:56:03.555961 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:03.555928 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:56:03.676542 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:03.676500 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0245bb0d-xtfvr\" (UID: \"479f2e22-3f3e-40f8-8fb6-3b98d34373c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0245bb0d-xtfvr" Nov 02 18:56:03.676754 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:03.676637 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbt9v\" (UniqueName: \"kubernetes.io/projected/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-kube-api-access-zbt9v\") pod \"maas-api-revoke-sub-0245bb0d-xtfvr\" (UID: \"479f2e22-3f3e-40f8-8fb6-3b98d34373c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0245bb0d-xtfvr" Nov 02 18:56:03.676754 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:03.676717 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0245bb0d-xtfvr\" (UID: \"479f2e22-3f3e-40f8-8fb6-3b98d34373c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0245bb0d-xtfvr" Nov 02 18:56:03.777491 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:03.777456 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0245bb0d-xtfvr\" (UID: \"479f2e22-3f3e-40f8-8fb6-3b98d34373c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0245bb0d-xtfvr" Nov 02 18:56:03.777717 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:03.777516 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbt9v\" (UniqueName: \"kubernetes.io/projected/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-kube-api-access-zbt9v\") pod \"maas-api-revoke-sub-0245bb0d-xtfvr\" (UID: \"479f2e22-3f3e-40f8-8fb6-3b98d34373c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0245bb0d-xtfvr" Nov 02 18:56:03.777717 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:03.777694 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0245bb0d-xtfvr\" (UID: \"479f2e22-3f3e-40f8-8fb6-3b98d34373c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0245bb0d-xtfvr" Nov 02 18:56:03.778194 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:03.778167 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0245bb0d-xtfvr\" (UID: \"479f2e22-3f3e-40f8-8fb6-3b98d34373c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0245bb0d-xtfvr" Nov 02 18:56:03.780044 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:03.780023 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0245bb0d-xtfvr\" (UID: \"479f2e22-3f3e-40f8-8fb6-3b98d34373c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0245bb0d-xtfvr" Nov 02 18:56:03.785675 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:03.785657 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbt9v\" (UniqueName: \"kubernetes.io/projected/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-kube-api-access-zbt9v\") pod \"maas-api-revoke-sub-0245bb0d-xtfvr\" (UID: \"479f2e22-3f3e-40f8-8fb6-3b98d34373c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0245bb0d-xtfvr" Nov 02 18:56:03.863837 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:03.863779 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0245bb0d-xtfvr" Nov 02 18:56:04.137784 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:04.137753 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0245bb0d-xtfvr"] Nov 02 18:56:04.139724 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:56:04.139680 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479f2e22_3f3e_40f8_8fb6_3b98d34373c3.slice/crio-e459d3c8a8a74ac0cb51a00b8f1206c7ef6df04b0047e5c45662f553a291e181 WatchSource:0}: Error finding container e459d3c8a8a74ac0cb51a00b8f1206c7ef6df04b0047e5c45662f553a291e181: Status 404 returned error can't find the container with id e459d3c8a8a74ac0cb51a00b8f1206c7ef6df04b0047e5c45662f553a291e181 Nov 02 18:56:04.224390 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:04.224359 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0245bb0d-xtfvr" event={"ID":"479f2e22-3f3e-40f8-8fb6-3b98d34373c3","Type":"ContainerStarted","Data":"e459d3c8a8a74ac0cb51a00b8f1206c7ef6df04b0047e5c45662f553a291e181"} Nov 02 18:56:04.481207 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:56:04.481176 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:56:05.233911 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:05.233803 5601 generic.go:356] "Generic (PLEG): container finished" podID="479f2e22-3f3e-40f8-8fb6-3b98d34373c3" containerID="64141e9d2c9e1d4b77f8616e509fde3175faf9b5eb448b05b68af85ae7b24d9a" exitCode=0 Nov 02 18:56:05.233911 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:05.233847 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0245bb0d-xtfvr" event={"ID":"479f2e22-3f3e-40f8-8fb6-3b98d34373c3","Type":"ContainerDied","Data":"64141e9d2c9e1d4b77f8616e509fde3175faf9b5eb448b05b68af85ae7b24d9a"} Nov 02 18:56:06.594883 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:06.594818 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0245bb0d-xtfvr" Nov 02 18:56:06.618114 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:06.618088 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-maas-api-openshift-ca\") pod \"479f2e22-3f3e-40f8-8fb6-3b98d34373c3\" (UID: \"479f2e22-3f3e-40f8-8fb6-3b98d34373c3\") " Nov 02 18:56:06.618254 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:06.618138 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-maas-api-xks-ca\") pod \"479f2e22-3f3e-40f8-8fb6-3b98d34373c3\" (UID: \"479f2e22-3f3e-40f8-8fb6-3b98d34373c3\") " Nov 02 18:56:06.618254 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:06.618161 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-kube-api-access-zbt9v\" (UniqueName: \"kubernetes.io/projected/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-kube-api-access-zbt9v\") pod \"479f2e22-3f3e-40f8-8fb6-3b98d34373c3\" (UID: \"479f2e22-3f3e-40f8-8fb6-3b98d34373c3\") " Nov 02 18:56:06.618567 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:06.618548 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-maas-api-openshift-ca" pod "479f2e22-3f3e-40f8-8fb6-3b98d34373c3" (UID: "479f2e22-3f3e-40f8-8fb6-3b98d34373c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:56:06.620262 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:06.620235 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-maas-api-xks-ca" pod "479f2e22-3f3e-40f8-8fb6-3b98d34373c3" (UID: "479f2e22-3f3e-40f8-8fb6-3b98d34373c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:56:06.620363 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:06.620259 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-kube-api-access-zbt9v" pod "479f2e22-3f3e-40f8-8fb6-3b98d34373c3" (UID: "479f2e22-3f3e-40f8-8fb6-3b98d34373c3"). InnerVolumeSpecName "kube-api-access-zbt9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:56:06.718827 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:06.718784 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:56:06.718827 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:06.718828 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:56:06.719033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:06.718840 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbt9v\" (UniqueName: \"kubernetes.io/projected/479f2e22-3f3e-40f8-8fb6-3b98d34373c3-kube-api-access-zbt9v\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:56:07.270876 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:07.270770 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0245bb0d-xtfvr" event={"ID":"479f2e22-3f3e-40f8-8fb6-3b98d34373c3","Type":"ContainerDied","Data":"e459d3c8a8a74ac0cb51a00b8f1206c7ef6df04b0047e5c45662f553a291e181"} Nov 02 18:56:07.271202 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:07.271125 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e459d3c8a8a74ac0cb51a00b8f1206c7ef6df04b0047e5c45662f553a291e181" Nov 02 18:56:07.271402 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:07.271362 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0245bb0d-xtfvr" Nov 02 18:56:13.741449 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:13.741411 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88c5abea-vl9v2"] Nov 02 18:56:13.741994 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:13.741971 5601 cpu_manager.go:414] "containerMap: removing container" podUID="479f2e22-3f3e-40f8-8fb6-3b98d34373c3" containerName="revoke-keys" Nov 02 18:56:13.742075 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:13.741997 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="479f2e22-3f3e-40f8-8fb6-3b98d34373c3" containerName="revoke-keys" Nov 02 18:56:13.742171 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:13.742158 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="479f2e22-3f3e-40f8-8fb6-3b98d34373c3" containerName="revoke-keys" Nov 02 18:56:13.745426 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:13.745391 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c5abea-vl9v2" Nov 02 18:56:13.751601 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:13.751575 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:56:13.752242 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:13.752218 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:56:13.753219 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:13.753192 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88c5abea-vl9v2"] Nov 02 18:56:13.769547 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:13.769516 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv7kt\" (UniqueName: \"kubernetes.io/projected/02762677-23a1-4a96-b27f-036a2baf746c-kube-api-access-vv7kt\") pod \"maas-api-revoke-sub-88c5abea-vl9v2\" (UID: \"02762677-23a1-4a96-b27f-036a2baf746c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c5abea-vl9v2" Nov 02 18:56:13.769715 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:13.769644 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02762677-23a1-4a96-b27f-036a2baf746c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88c5abea-vl9v2\" (UID: \"02762677-23a1-4a96-b27f-036a2baf746c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c5abea-vl9v2" Nov 02 18:56:13.769715 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:13.769700 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02762677-23a1-4a96-b27f-036a2baf746c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88c5abea-vl9v2\" (UID: \"02762677-23a1-4a96-b27f-036a2baf746c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c5abea-vl9v2" Nov 02 18:56:13.871037 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:13.871001 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv7kt\" (UniqueName: \"kubernetes.io/projected/02762677-23a1-4a96-b27f-036a2baf746c-kube-api-access-vv7kt\") pod \"maas-api-revoke-sub-88c5abea-vl9v2\" (UID: \"02762677-23a1-4a96-b27f-036a2baf746c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c5abea-vl9v2" Nov 02 18:56:13.871219 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:13.871102 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02762677-23a1-4a96-b27f-036a2baf746c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88c5abea-vl9v2\" (UID: \"02762677-23a1-4a96-b27f-036a2baf746c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c5abea-vl9v2" Nov 02 18:56:13.871219 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:13.871157 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02762677-23a1-4a96-b27f-036a2baf746c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88c5abea-vl9v2\" (UID: \"02762677-23a1-4a96-b27f-036a2baf746c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c5abea-vl9v2" Nov 02 18:56:13.871947 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:13.871927 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02762677-23a1-4a96-b27f-036a2baf746c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88c5abea-vl9v2\" (UID: \"02762677-23a1-4a96-b27f-036a2baf746c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c5abea-vl9v2" Nov 02 18:56:13.873585 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:13.873565 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02762677-23a1-4a96-b27f-036a2baf746c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88c5abea-vl9v2\" (UID: \"02762677-23a1-4a96-b27f-036a2baf746c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c5abea-vl9v2" Nov 02 18:56:13.880160 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:13.880138 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv7kt\" (UniqueName: \"kubernetes.io/projected/02762677-23a1-4a96-b27f-036a2baf746c-kube-api-access-vv7kt\") pod \"maas-api-revoke-sub-88c5abea-vl9v2\" (UID: \"02762677-23a1-4a96-b27f-036a2baf746c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c5abea-vl9v2" Nov 02 18:56:14.061112 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:14.061029 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c5abea-vl9v2" Nov 02 18:56:14.332628 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:14.332578 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88c5abea-vl9v2"] Nov 02 18:56:14.334207 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:56:14.334180 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02762677_23a1_4a96_b27f_036a2baf746c.slice/crio-e8d175690c025438a294559b78d0518bf3dd6d836f82275063e091f2ebe711f8 WatchSource:0}: Error finding container e8d175690c025438a294559b78d0518bf3dd6d836f82275063e091f2ebe711f8: Status 404 returned error can't find the container with id e8d175690c025438a294559b78d0518bf3dd6d836f82275063e091f2ebe711f8 Nov 02 18:56:15.312734 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:15.312700 5601 generic.go:356] "Generic (PLEG): container finished" podID="02762677-23a1-4a96-b27f-036a2baf746c" containerID="8cf84e2ada3551b902a5d6ed15ecff6238607ec0c79d0569e70d1c0fb25c3a98" exitCode=0 Nov 02 18:56:15.313130 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:15.312750 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c5abea-vl9v2" event={"ID":"02762677-23a1-4a96-b27f-036a2baf746c","Type":"ContainerDied","Data":"8cf84e2ada3551b902a5d6ed15ecff6238607ec0c79d0569e70d1c0fb25c3a98"} Nov 02 18:56:15.313130 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:15.312792 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c5abea-vl9v2" event={"ID":"02762677-23a1-4a96-b27f-036a2baf746c","Type":"ContainerStarted","Data":"e8d175690c025438a294559b78d0518bf3dd6d836f82275063e091f2ebe711f8"} Nov 02 18:56:16.449337 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:16.449311 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c5abea-vl9v2" Nov 02 18:56:16.498789 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:16.498752 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02762677-23a1-4a96-b27f-036a2baf746c-kube-api-access-vv7kt\" (UniqueName: \"kubernetes.io/projected/02762677-23a1-4a96-b27f-036a2baf746c-kube-api-access-vv7kt\") pod \"02762677-23a1-4a96-b27f-036a2baf746c\" (UID: \"02762677-23a1-4a96-b27f-036a2baf746c\") " Nov 02 18:56:16.498970 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:16.498839 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02762677-23a1-4a96-b27f-036a2baf746c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02762677-23a1-4a96-b27f-036a2baf746c-maas-api-openshift-ca\") pod \"02762677-23a1-4a96-b27f-036a2baf746c\" (UID: \"02762677-23a1-4a96-b27f-036a2baf746c\") " Nov 02 18:56:16.498970 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:16.498885 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02762677-23a1-4a96-b27f-036a2baf746c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02762677-23a1-4a96-b27f-036a2baf746c-maas-api-xks-ca\") pod \"02762677-23a1-4a96-b27f-036a2baf746c\" (UID: \"02762677-23a1-4a96-b27f-036a2baf746c\") " Nov 02 18:56:16.499289 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:16.499256 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02762677-23a1-4a96-b27f-036a2baf746c-maas-api-openshift-ca" pod "02762677-23a1-4a96-b27f-036a2baf746c" (UID: "02762677-23a1-4a96-b27f-036a2baf746c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:56:16.501164 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:16.501114 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02762677-23a1-4a96-b27f-036a2baf746c-maas-api-xks-ca" pod "02762677-23a1-4a96-b27f-036a2baf746c" (UID: "02762677-23a1-4a96-b27f-036a2baf746c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:56:16.501277 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:16.501173 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02762677-23a1-4a96-b27f-036a2baf746c-kube-api-access-vv7kt" pod "02762677-23a1-4a96-b27f-036a2baf746c" (UID: "02762677-23a1-4a96-b27f-036a2baf746c"). InnerVolumeSpecName "kube-api-access-vv7kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:56:16.600142 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:16.600051 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv7kt\" (UniqueName: \"kubernetes.io/projected/02762677-23a1-4a96-b27f-036a2baf746c-kube-api-access-vv7kt\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:56:16.600142 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:16.600086 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02762677-23a1-4a96-b27f-036a2baf746c-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:56:16.600142 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:16.600099 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02762677-23a1-4a96-b27f-036a2baf746c-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:56:17.324997 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:17.324961 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c5abea-vl9v2" Nov 02 18:56:17.324997 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:17.324982 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c5abea-vl9v2" event={"ID":"02762677-23a1-4a96-b27f-036a2baf746c","Type":"ContainerDied","Data":"e8d175690c025438a294559b78d0518bf3dd6d836f82275063e091f2ebe711f8"} Nov 02 18:56:17.325233 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:17.325073 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8d175690c025438a294559b78d0518bf3dd6d836f82275063e091f2ebe711f8" Nov 02 18:56:23.965605 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:23.965559 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25"] Nov 02 18:56:23.966023 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:23.965994 5601 cpu_manager.go:414] "containerMap: removing container" podUID="02762677-23a1-4a96-b27f-036a2baf746c" containerName="revoke-keys" Nov 02 18:56:23.966023 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:23.966007 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02762677-23a1-4a96-b27f-036a2baf746c" containerName="revoke-keys" Nov 02 18:56:23.966118 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:23.966109 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="02762677-23a1-4a96-b27f-036a2baf746c" containerName="revoke-keys" Nov 02 18:56:23.969601 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:23.969582 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25" Nov 02 18:56:23.973914 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:23.973878 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:56:23.974430 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:23.974404 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:56:23.976051 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:23.976020 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25"] Nov 02 18:56:24.064471 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:24.064427 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e037e2db-712d-41a3-bff7-1bae2fc63897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a524851-75m25\" (UID: \"e037e2db-712d-41a3-bff7-1bae2fc63897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25" Nov 02 18:56:24.064679 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:24.064474 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e037e2db-712d-41a3-bff7-1bae2fc63897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a524851-75m25\" (UID: \"e037e2db-712d-41a3-bff7-1bae2fc63897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25" Nov 02 18:56:24.064679 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:24.064634 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2jrg\" (UniqueName: \"kubernetes.io/projected/e037e2db-712d-41a3-bff7-1bae2fc63897-kube-api-access-m2jrg\") pod \"maas-api-revoke-sub-9a524851-75m25\" (UID: \"e037e2db-712d-41a3-bff7-1bae2fc63897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25" Nov 02 18:56:24.165758 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:24.165721 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e037e2db-712d-41a3-bff7-1bae2fc63897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a524851-75m25\" (UID: \"e037e2db-712d-41a3-bff7-1bae2fc63897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25" Nov 02 18:56:24.165949 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:24.165818 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2jrg\" (UniqueName: \"kubernetes.io/projected/e037e2db-712d-41a3-bff7-1bae2fc63897-kube-api-access-m2jrg\") pod \"maas-api-revoke-sub-9a524851-75m25\" (UID: \"e037e2db-712d-41a3-bff7-1bae2fc63897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25" Nov 02 18:56:24.165949 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:24.165871 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e037e2db-712d-41a3-bff7-1bae2fc63897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a524851-75m25\" (UID: \"e037e2db-712d-41a3-bff7-1bae2fc63897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25" Nov 02 18:56:24.166417 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:24.166382 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e037e2db-712d-41a3-bff7-1bae2fc63897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a524851-75m25\" (UID: \"e037e2db-712d-41a3-bff7-1bae2fc63897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25" Nov 02 18:56:24.168243 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:24.168219 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e037e2db-712d-41a3-bff7-1bae2fc63897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a524851-75m25\" (UID: \"e037e2db-712d-41a3-bff7-1bae2fc63897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25" Nov 02 18:56:24.174829 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:24.174794 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2jrg\" (UniqueName: \"kubernetes.io/projected/e037e2db-712d-41a3-bff7-1bae2fc63897-kube-api-access-m2jrg\") pod \"maas-api-revoke-sub-9a524851-75m25\" (UID: \"e037e2db-712d-41a3-bff7-1bae2fc63897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25" Nov 02 18:56:24.286513 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:24.286429 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25" Nov 02 18:56:24.563118 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:24.563085 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25"] Nov 02 18:56:24.564923 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:56:24.564894 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode037e2db_712d_41a3_bff7_1bae2fc63897.slice/crio-b58169092240bafd6a6e41d83c87f3cb22d24ad2bca19ecdec51a3cf3a3270c2 WatchSource:0}: Error finding container b58169092240bafd6a6e41d83c87f3cb22d24ad2bca19ecdec51a3cf3a3270c2: Status 404 returned error can't find the container with id b58169092240bafd6a6e41d83c87f3cb22d24ad2bca19ecdec51a3cf3a3270c2 Nov 02 18:56:25.369682 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:25.369645 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25" event={"ID":"e037e2db-712d-41a3-bff7-1bae2fc63897","Type":"ContainerStarted","Data":"912c54c8b12720277d098afdb9f0ae0b7576ff075fbc4daae6b585ddb8145544"} Nov 02 18:56:25.370105 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:25.369704 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25" event={"ID":"e037e2db-712d-41a3-bff7-1bae2fc63897","Type":"ContainerStarted","Data":"b58169092240bafd6a6e41d83c87f3cb22d24ad2bca19ecdec51a3cf3a3270c2"} Nov 02 18:56:25.386811 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:25.386765 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25" podStartSLOduration=2.386753051 podStartE2EDuration="2.386753051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:56:25.385760989 +0000 UTC m=+4950.076815001" watchObservedRunningTime="2026-11-02 18:56:25.386753051 +0000 UTC m=+4950.077807062" Nov 02 18:56:26.375904 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:26.375869 5601 generic.go:356] "Generic (PLEG): container finished" podID="e037e2db-712d-41a3-bff7-1bae2fc63897" containerID="912c54c8b12720277d098afdb9f0ae0b7576ff075fbc4daae6b585ddb8145544" exitCode=0 Nov 02 18:56:26.376330 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:26.375914 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25" event={"ID":"e037e2db-712d-41a3-bff7-1bae2fc63897","Type":"ContainerDied","Data":"912c54c8b12720277d098afdb9f0ae0b7576ff075fbc4daae6b585ddb8145544"} Nov 02 18:56:27.520948 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:27.520920 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25" Nov 02 18:56:27.599416 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:27.599378 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e037e2db-712d-41a3-bff7-1bae2fc63897-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e037e2db-712d-41a3-bff7-1bae2fc63897-maas-api-xks-ca\") pod \"e037e2db-712d-41a3-bff7-1bae2fc63897\" (UID: \"e037e2db-712d-41a3-bff7-1bae2fc63897\") " Nov 02 18:56:27.599601 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:27.599474 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e037e2db-712d-41a3-bff7-1bae2fc63897-kube-api-access-m2jrg\" (UniqueName: \"kubernetes.io/projected/e037e2db-712d-41a3-bff7-1bae2fc63897-kube-api-access-m2jrg\") pod \"e037e2db-712d-41a3-bff7-1bae2fc63897\" (UID: \"e037e2db-712d-41a3-bff7-1bae2fc63897\") " Nov 02 18:56:27.599601 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:27.599547 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e037e2db-712d-41a3-bff7-1bae2fc63897-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e037e2db-712d-41a3-bff7-1bae2fc63897-maas-api-openshift-ca\") pod \"e037e2db-712d-41a3-bff7-1bae2fc63897\" (UID: \"e037e2db-712d-41a3-bff7-1bae2fc63897\") " Nov 02 18:56:27.599874 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:27.599850 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e037e2db-712d-41a3-bff7-1bae2fc63897-maas-api-openshift-ca" pod "e037e2db-712d-41a3-bff7-1bae2fc63897" (UID: "e037e2db-712d-41a3-bff7-1bae2fc63897"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:56:27.601537 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:27.601507 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e037e2db-712d-41a3-bff7-1bae2fc63897-maas-api-xks-ca" pod "e037e2db-712d-41a3-bff7-1bae2fc63897" (UID: "e037e2db-712d-41a3-bff7-1bae2fc63897"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:56:27.601661 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:27.601561 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e037e2db-712d-41a3-bff7-1bae2fc63897-kube-api-access-m2jrg" pod "e037e2db-712d-41a3-bff7-1bae2fc63897" (UID: "e037e2db-712d-41a3-bff7-1bae2fc63897"). InnerVolumeSpecName "kube-api-access-m2jrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:56:27.701199 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:27.701164 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2jrg\" (UniqueName: \"kubernetes.io/projected/e037e2db-712d-41a3-bff7-1bae2fc63897-kube-api-access-m2jrg\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:56:27.701199 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:27.701195 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e037e2db-712d-41a3-bff7-1bae2fc63897-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:56:27.701402 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:27.701206 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e037e2db-712d-41a3-bff7-1bae2fc63897-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:56:28.387330 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:28.387296 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25" Nov 02 18:56:28.387514 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:28.387331 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a524851-75m25" event={"ID":"e037e2db-712d-41a3-bff7-1bae2fc63897","Type":"ContainerDied","Data":"b58169092240bafd6a6e41d83c87f3cb22d24ad2bca19ecdec51a3cf3a3270c2"} Nov 02 18:56:28.387514 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:28.387379 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b58169092240bafd6a6e41d83c87f3cb22d24ad2bca19ecdec51a3cf3a3270c2" Nov 02 18:56:34.227320 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:34.227281 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5"] Nov 02 18:56:34.227851 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:34.227830 5601 cpu_manager.go:414] "containerMap: removing container" podUID="e037e2db-712d-41a3-bff7-1bae2fc63897" containerName="revoke-keys" Nov 02 18:56:34.227897 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:34.227852 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e037e2db-712d-41a3-bff7-1bae2fc63897" containerName="revoke-keys" Nov 02 18:56:34.227945 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:34.227935 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="e037e2db-712d-41a3-bff7-1bae2fc63897" containerName="revoke-keys" Nov 02 18:56:34.231255 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:34.231236 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5" Nov 02 18:56:34.235822 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:34.235793 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:56:34.235992 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:34.235867 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:56:34.237957 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:34.237930 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5"] Nov 02 18:56:34.363905 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:34.363856 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pbbh\" (UniqueName: \"kubernetes.io/projected/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-kube-api-access-6pbbh\") pod \"maas-api-revoke-sub-d762b3d1-297h5\" (UID: \"cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5" Nov 02 18:56:34.363905 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:34.363912 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d762b3d1-297h5\" (UID: \"cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5" Nov 02 18:56:34.364133 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:34.364026 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d762b3d1-297h5\" (UID: \"cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5" Nov 02 18:56:34.465409 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:34.465368 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pbbh\" (UniqueName: \"kubernetes.io/projected/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-kube-api-access-6pbbh\") pod \"maas-api-revoke-sub-d762b3d1-297h5\" (UID: \"cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5" Nov 02 18:56:34.465409 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:34.465410 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d762b3d1-297h5\" (UID: \"cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5" Nov 02 18:56:34.465716 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:34.465464 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d762b3d1-297h5\" (UID: \"cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5" Nov 02 18:56:34.466225 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:34.466195 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d762b3d1-297h5\" (UID: \"cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5" Nov 02 18:56:34.467857 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:34.467828 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d762b3d1-297h5\" (UID: \"cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5" Nov 02 18:56:34.473327 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:34.473294 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pbbh\" (UniqueName: \"kubernetes.io/projected/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-kube-api-access-6pbbh\") pod \"maas-api-revoke-sub-d762b3d1-297h5\" (UID: \"cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5" Nov 02 18:56:34.484647 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:56:34.484561 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:56:34.547629 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:34.547585 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5" Nov 02 18:56:34.820863 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:34.820837 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5"] Nov 02 18:56:34.822942 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:56:34.822905 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabbd5f1_2cc7_4be8_ace4_de85ca73bd0c.slice/crio-efe0b2c791c8943eea893d6e7ccbc39f47afb55ae6a6f0234bb2088583e47197 WatchSource:0}: Error finding container efe0b2c791c8943eea893d6e7ccbc39f47afb55ae6a6f0234bb2088583e47197: Status 404 returned error can't find the container with id efe0b2c791c8943eea893d6e7ccbc39f47afb55ae6a6f0234bb2088583e47197 Nov 02 18:56:35.430530 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:35.430493 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5" event={"ID":"cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c","Type":"ContainerStarted","Data":"228da02e4c64c0d526422d13d19d4d4e0776460c2d00c6e7226c4fbfa2f156e8"} Nov 02 18:56:35.430974 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:35.430559 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5" event={"ID":"cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c","Type":"ContainerStarted","Data":"efe0b2c791c8943eea893d6e7ccbc39f47afb55ae6a6f0234bb2088583e47197"} Nov 02 18:56:35.449024 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:35.448954 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5" podStartSLOduration=1.448932904 podStartE2EDuration="1.448932904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:56:35.446843479 +0000 UTC m=+4960.137897493" watchObservedRunningTime="2026-11-02 18:56:35.448932904 +0000 UTC m=+4960.139986914" Nov 02 18:56:36.455265 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:36.454539 5601 generic.go:356] "Generic (PLEG): container finished" podID="cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c" containerID="228da02e4c64c0d526422d13d19d4d4e0776460c2d00c6e7226c4fbfa2f156e8" exitCode=0 Nov 02 18:56:36.455265 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:36.454787 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5" event={"ID":"cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c","Type":"ContainerDied","Data":"228da02e4c64c0d526422d13d19d4d4e0776460c2d00c6e7226c4fbfa2f156e8"} Nov 02 18:56:37.730365 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:37.730309 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5" Nov 02 18:56:37.818975 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:37.818929 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-kube-api-access-6pbbh\" (UniqueName: \"kubernetes.io/projected/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-kube-api-access-6pbbh\") pod \"cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c\" (UID: \"cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c\") " Nov 02 18:56:37.819187 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:37.818994 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-maas-api-openshift-ca\") pod \"cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c\" (UID: \"cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c\") " Nov 02 18:56:37.819187 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:37.819032 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-maas-api-xks-ca\") pod \"cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c\" (UID: \"cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c\") " Nov 02 18:56:37.819538 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:37.819478 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-maas-api-openshift-ca" pod "cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c" (UID: "cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:56:37.821428 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:37.821404 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-maas-api-xks-ca" pod "cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c" (UID: "cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:56:37.821849 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:37.821825 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-kube-api-access-6pbbh" pod "cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c" (UID: "cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c"). InnerVolumeSpecName "kube-api-access-6pbbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:56:37.920050 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:37.920007 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pbbh\" (UniqueName: \"kubernetes.io/projected/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-kube-api-access-6pbbh\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:56:37.920050 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:37.920043 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:56:37.920050 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:37.920054 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:56:38.479562 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:38.479525 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5" event={"ID":"cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c","Type":"ContainerDied","Data":"efe0b2c791c8943eea893d6e7ccbc39f47afb55ae6a6f0234bb2088583e47197"} Nov 02 18:56:38.479784 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:38.479575 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efe0b2c791c8943eea893d6e7ccbc39f47afb55ae6a6f0234bb2088583e47197" Nov 02 18:56:38.479784 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:38.479572 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d762b3d1-297h5" Nov 02 18:56:44.430150 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:44.430110 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54"] Nov 02 18:56:44.430536 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:44.430496 5601 cpu_manager.go:414] "containerMap: removing container" podUID="cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c" containerName="revoke-keys" Nov 02 18:56:44.430536 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:44.430506 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c" containerName="revoke-keys" Nov 02 18:56:44.430646 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:44.430574 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c" containerName="revoke-keys" Nov 02 18:56:44.433611 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:44.433589 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54" Nov 02 18:56:44.439120 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:44.439078 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:56:44.439120 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:44.439078 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:56:44.442311 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:44.442282 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54"] Nov 02 18:56:44.483296 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:44.483260 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b3208e8-h9v54\" (UID: \"d3a14d7d-9aba-4dd2-af15-5a5853c5cf04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54" Nov 02 18:56:44.483471 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:44.483352 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b3208e8-h9v54\" (UID: \"d3a14d7d-9aba-4dd2-af15-5a5853c5cf04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54" Nov 02 18:56:44.483519 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:44.483494 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jdz7\" (UniqueName: \"kubernetes.io/projected/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-kube-api-access-9jdz7\") pod \"maas-api-revoke-sub-1b3208e8-h9v54\" (UID: \"d3a14d7d-9aba-4dd2-af15-5a5853c5cf04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54" Nov 02 18:56:44.585020 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:44.584981 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b3208e8-h9v54\" (UID: \"d3a14d7d-9aba-4dd2-af15-5a5853c5cf04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54" Nov 02 18:56:44.585203 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:44.585029 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jdz7\" (UniqueName: \"kubernetes.io/projected/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-kube-api-access-9jdz7\") pod \"maas-api-revoke-sub-1b3208e8-h9v54\" (UID: \"d3a14d7d-9aba-4dd2-af15-5a5853c5cf04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54" Nov 02 18:56:44.585248 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:44.585215 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b3208e8-h9v54\" (UID: \"d3a14d7d-9aba-4dd2-af15-5a5853c5cf04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54" Nov 02 18:56:44.585683 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:44.585658 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b3208e8-h9v54\" (UID: \"d3a14d7d-9aba-4dd2-af15-5a5853c5cf04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54" Nov 02 18:56:44.587645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:44.587605 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b3208e8-h9v54\" (UID: \"d3a14d7d-9aba-4dd2-af15-5a5853c5cf04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54" Nov 02 18:56:44.593353 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:44.593323 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jdz7\" (UniqueName: \"kubernetes.io/projected/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-kube-api-access-9jdz7\") pod \"maas-api-revoke-sub-1b3208e8-h9v54\" (UID: \"d3a14d7d-9aba-4dd2-af15-5a5853c5cf04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54" Nov 02 18:56:44.749025 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:44.748948 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54" Nov 02 18:56:45.023734 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:45.023703 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54"] Nov 02 18:56:45.025305 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:56:45.025278 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a14d7d_9aba_4dd2_af15_5a5853c5cf04.slice/crio-21747f54fe25ba3c785f5e321df7913b28f118e4a7374d91c7eb30f857d68910 WatchSource:0}: Error finding container 21747f54fe25ba3c785f5e321df7913b28f118e4a7374d91c7eb30f857d68910: Status 404 returned error can't find the container with id 21747f54fe25ba3c785f5e321df7913b28f118e4a7374d91c7eb30f857d68910 Nov 02 18:56:45.521125 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:45.521083 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54" event={"ID":"d3a14d7d-9aba-4dd2-af15-5a5853c5cf04","Type":"ContainerStarted","Data":"ac562d005e2eff62ac498c6661ebfb2467c1f465dfba4a94bfd4b4bbf3f1d64f"} Nov 02 18:56:45.521740 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:45.521706 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54" event={"ID":"d3a14d7d-9aba-4dd2-af15-5a5853c5cf04","Type":"ContainerStarted","Data":"21747f54fe25ba3c785f5e321df7913b28f118e4a7374d91c7eb30f857d68910"} Nov 02 18:56:45.541233 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:45.541180 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54" podStartSLOduration=1.541158175 podStartE2EDuration="1.541158175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:56:45.537732358 +0000 UTC m=+4970.228786371" watchObservedRunningTime="2026-11-02 18:56:45.541158175 +0000 UTC m=+4970.232212190" Nov 02 18:56:46.528632 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:46.528579 5601 generic.go:356] "Generic (PLEG): container finished" podID="d3a14d7d-9aba-4dd2-af15-5a5853c5cf04" containerID="ac562d005e2eff62ac498c6661ebfb2467c1f465dfba4a94bfd4b4bbf3f1d64f" exitCode=0 Nov 02 18:56:46.529081 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:46.528671 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54" event={"ID":"d3a14d7d-9aba-4dd2-af15-5a5853c5cf04","Type":"ContainerDied","Data":"ac562d005e2eff62ac498c6661ebfb2467c1f465dfba4a94bfd4b4bbf3f1d64f"} Nov 02 18:56:47.921516 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:47.921421 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54" Nov 02 18:56:47.926407 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:47.926374 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-kube-api-access-9jdz7\" (UniqueName: \"kubernetes.io/projected/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-kube-api-access-9jdz7\") pod \"d3a14d7d-9aba-4dd2-af15-5a5853c5cf04\" (UID: \"d3a14d7d-9aba-4dd2-af15-5a5853c5cf04\") " Nov 02 18:56:47.926554 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:47.926431 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-maas-api-xks-ca\") pod \"d3a14d7d-9aba-4dd2-af15-5a5853c5cf04\" (UID: \"d3a14d7d-9aba-4dd2-af15-5a5853c5cf04\") " Nov 02 18:56:47.926554 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:47.926498 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-maas-api-openshift-ca\") pod \"d3a14d7d-9aba-4dd2-af15-5a5853c5cf04\" (UID: \"d3a14d7d-9aba-4dd2-af15-5a5853c5cf04\") " Nov 02 18:56:47.927099 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:47.927076 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-maas-api-openshift-ca" pod "d3a14d7d-9aba-4dd2-af15-5a5853c5cf04" (UID: "d3a14d7d-9aba-4dd2-af15-5a5853c5cf04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:56:47.928694 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:47.928669 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-maas-api-xks-ca" pod "d3a14d7d-9aba-4dd2-af15-5a5853c5cf04" (UID: "d3a14d7d-9aba-4dd2-af15-5a5853c5cf04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:56:47.929052 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:47.929027 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-kube-api-access-9jdz7" pod "d3a14d7d-9aba-4dd2-af15-5a5853c5cf04" (UID: "d3a14d7d-9aba-4dd2-af15-5a5853c5cf04"). InnerVolumeSpecName "kube-api-access-9jdz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:56:48.027380 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:48.027350 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jdz7\" (UniqueName: \"kubernetes.io/projected/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-kube-api-access-9jdz7\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:56:48.027380 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:48.027379 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:56:48.027589 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:48.027391 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3a14d7d-9aba-4dd2-af15-5a5853c5cf04-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:56:48.540302 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:48.540272 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54" Nov 02 18:56:48.540503 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:48.540302 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3208e8-h9v54" event={"ID":"d3a14d7d-9aba-4dd2-af15-5a5853c5cf04","Type":"ContainerDied","Data":"21747f54fe25ba3c785f5e321df7913b28f118e4a7374d91c7eb30f857d68910"} Nov 02 18:56:48.540503 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:48.540353 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21747f54fe25ba3c785f5e321df7913b28f118e4a7374d91c7eb30f857d68910" Nov 02 18:56:54.627796 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:54.627762 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb"] Nov 02 18:56:54.628219 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:54.628154 5601 cpu_manager.go:414] "containerMap: removing container" podUID="d3a14d7d-9aba-4dd2-af15-5a5853c5cf04" containerName="revoke-keys" Nov 02 18:56:54.628219 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:54.628165 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a14d7d-9aba-4dd2-af15-5a5853c5cf04" containerName="revoke-keys" Nov 02 18:56:54.628305 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:54.628250 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a14d7d-9aba-4dd2-af15-5a5853c5cf04" containerName="revoke-keys" Nov 02 18:56:54.631808 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:54.631789 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb" Nov 02 18:56:54.639331 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:54.639190 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:56:54.639331 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:54.639321 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:56:54.641717 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:54.641686 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb"] Nov 02 18:56:54.690056 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:54.690021 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86b7d192-19fa-4e37-a839-dc373a1774b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9d0f7b3-5gxqb\" (UID: \"86b7d192-19fa-4e37-a839-dc373a1774b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb" Nov 02 18:56:54.690220 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:54.690062 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nxbb\" (UniqueName: \"kubernetes.io/projected/86b7d192-19fa-4e37-a839-dc373a1774b7-kube-api-access-2nxbb\") pod \"maas-api-revoke-sub-f9d0f7b3-5gxqb\" (UID: \"86b7d192-19fa-4e37-a839-dc373a1774b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb" Nov 02 18:56:54.690220 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:54.690198 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86b7d192-19fa-4e37-a839-dc373a1774b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9d0f7b3-5gxqb\" (UID: \"86b7d192-19fa-4e37-a839-dc373a1774b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb" Nov 02 18:56:54.791722 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:54.791674 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86b7d192-19fa-4e37-a839-dc373a1774b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9d0f7b3-5gxqb\" (UID: \"86b7d192-19fa-4e37-a839-dc373a1774b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb" Nov 02 18:56:54.791993 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:54.791784 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86b7d192-19fa-4e37-a839-dc373a1774b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9d0f7b3-5gxqb\" (UID: \"86b7d192-19fa-4e37-a839-dc373a1774b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb" Nov 02 18:56:54.791993 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:54.791816 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nxbb\" (UniqueName: \"kubernetes.io/projected/86b7d192-19fa-4e37-a839-dc373a1774b7-kube-api-access-2nxbb\") pod \"maas-api-revoke-sub-f9d0f7b3-5gxqb\" (UID: \"86b7d192-19fa-4e37-a839-dc373a1774b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb" Nov 02 18:56:54.792469 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:54.792443 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86b7d192-19fa-4e37-a839-dc373a1774b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9d0f7b3-5gxqb\" (UID: \"86b7d192-19fa-4e37-a839-dc373a1774b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb" Nov 02 18:56:54.794184 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:54.794163 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86b7d192-19fa-4e37-a839-dc373a1774b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9d0f7b3-5gxqb\" (UID: \"86b7d192-19fa-4e37-a839-dc373a1774b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb" Nov 02 18:56:54.799833 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:54.799811 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nxbb\" (UniqueName: \"kubernetes.io/projected/86b7d192-19fa-4e37-a839-dc373a1774b7-kube-api-access-2nxbb\") pod \"maas-api-revoke-sub-f9d0f7b3-5gxqb\" (UID: \"86b7d192-19fa-4e37-a839-dc373a1774b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb" Nov 02 18:56:54.946596 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:54.946563 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb" Nov 02 18:56:55.222062 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:55.222032 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb"] Nov 02 18:56:55.223755 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:56:55.223728 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b7d192_19fa_4e37_a839_dc373a1774b7.slice/crio-1340d8fe57927a02e8c06879cd8890dbb56f417ff2e5ccfc6a1914747605664c WatchSource:0}: Error finding container 1340d8fe57927a02e8c06879cd8890dbb56f417ff2e5ccfc6a1914747605664c: Status 404 returned error can't find the container with id 1340d8fe57927a02e8c06879cd8890dbb56f417ff2e5ccfc6a1914747605664c Nov 02 18:56:55.578372 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:55.578331 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb" event={"ID":"86b7d192-19fa-4e37-a839-dc373a1774b7","Type":"ContainerStarted","Data":"fccf02560be8f5d86222f4db47a6cf4432c7d6a20996f64e46f1d641e9adae7d"} Nov 02 18:56:55.578583 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:55.578392 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb" event={"ID":"86b7d192-19fa-4e37-a839-dc373a1774b7","Type":"ContainerStarted","Data":"1340d8fe57927a02e8c06879cd8890dbb56f417ff2e5ccfc6a1914747605664c"} Nov 02 18:56:55.595390 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:55.595341 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb" podStartSLOduration=1.595327459 podStartE2EDuration="1.595327459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:56:55.59384051 +0000 UTC m=+4980.284894554" watchObservedRunningTime="2026-11-02 18:56:55.595327459 +0000 UTC m=+4980.286381470" Nov 02 18:56:56.611409 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:56.611364 5601 generic.go:356] "Generic (PLEG): container finished" podID="86b7d192-19fa-4e37-a839-dc373a1774b7" containerID="fccf02560be8f5d86222f4db47a6cf4432c7d6a20996f64e46f1d641e9adae7d" exitCode=0 Nov 02 18:56:56.612038 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:56.611498 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb" event={"ID":"86b7d192-19fa-4e37-a839-dc373a1774b7","Type":"ContainerDied","Data":"fccf02560be8f5d86222f4db47a6cf4432c7d6a20996f64e46f1d641e9adae7d"} Nov 02 18:56:57.974639 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:57.968597 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb" Nov 02 18:56:58.030807 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:58.030776 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86b7d192-19fa-4e37-a839-dc373a1774b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86b7d192-19fa-4e37-a839-dc373a1774b7-maas-api-xks-ca\") pod \"86b7d192-19fa-4e37-a839-dc373a1774b7\" (UID: \"86b7d192-19fa-4e37-a839-dc373a1774b7\") " Nov 02 18:56:58.031024 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:58.030876 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b7d192-19fa-4e37-a839-dc373a1774b7-kube-api-access-2nxbb\" (UniqueName: \"kubernetes.io/projected/86b7d192-19fa-4e37-a839-dc373a1774b7-kube-api-access-2nxbb\") pod \"86b7d192-19fa-4e37-a839-dc373a1774b7\" (UID: \"86b7d192-19fa-4e37-a839-dc373a1774b7\") " Nov 02 18:56:58.031024 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:58.030932 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86b7d192-19fa-4e37-a839-dc373a1774b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86b7d192-19fa-4e37-a839-dc373a1774b7-maas-api-openshift-ca\") pod \"86b7d192-19fa-4e37-a839-dc373a1774b7\" (UID: \"86b7d192-19fa-4e37-a839-dc373a1774b7\") " Nov 02 18:56:58.031321 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:58.031287 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b7d192-19fa-4e37-a839-dc373a1774b7-maas-api-openshift-ca" pod "86b7d192-19fa-4e37-a839-dc373a1774b7" (UID: "86b7d192-19fa-4e37-a839-dc373a1774b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:56:58.032779 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:58.032744 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86b7d192-19fa-4e37-a839-dc373a1774b7-maas-api-xks-ca" pod "86b7d192-19fa-4e37-a839-dc373a1774b7" (UID: "86b7d192-19fa-4e37-a839-dc373a1774b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:56:58.032888 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:58.032862 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b7d192-19fa-4e37-a839-dc373a1774b7-kube-api-access-2nxbb" pod "86b7d192-19fa-4e37-a839-dc373a1774b7" (UID: "86b7d192-19fa-4e37-a839-dc373a1774b7"). InnerVolumeSpecName "kube-api-access-2nxbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:56:58.132033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:58.131978 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86b7d192-19fa-4e37-a839-dc373a1774b7-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:56:58.132033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:58.132030 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86b7d192-19fa-4e37-a839-dc373a1774b7-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:56:58.132033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:58.132043 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nxbb\" (UniqueName: \"kubernetes.io/projected/86b7d192-19fa-4e37-a839-dc373a1774b7-kube-api-access-2nxbb\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:56:58.624091 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:58.624062 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb" Nov 02 18:56:58.624278 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:58.624088 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d0f7b3-5gxqb" event={"ID":"86b7d192-19fa-4e37-a839-dc373a1774b7","Type":"ContainerDied","Data":"1340d8fe57927a02e8c06879cd8890dbb56f417ff2e5ccfc6a1914747605664c"} Nov 02 18:56:58.624278 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:56:58.624132 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1340d8fe57927a02e8c06879cd8890dbb56f417ff2e5ccfc6a1914747605664c" Nov 02 18:57:04.485070 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:57:04.485023 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:57:04.858881 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:04.858799 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc52e205-gdnrw"] Nov 02 18:57:04.859209 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:04.859196 5601 cpu_manager.go:414] "containerMap: removing container" podUID="86b7d192-19fa-4e37-a839-dc373a1774b7" containerName="revoke-keys" Nov 02 18:57:04.859269 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:04.859210 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b7d192-19fa-4e37-a839-dc373a1774b7" containerName="revoke-keys" Nov 02 18:57:04.859308 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:04.859272 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b7d192-19fa-4e37-a839-dc373a1774b7" containerName="revoke-keys" Nov 02 18:57:04.862981 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:04.862952 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc52e205-gdnrw" Nov 02 18:57:04.868259 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:04.868213 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:57:04.868259 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:04.868252 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:57:04.871255 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:04.871218 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc52e205-gdnrw"] Nov 02 18:57:04.890907 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:04.890867 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx4kr\" (UniqueName: \"kubernetes.io/projected/3b0daf67-c56b-4024-bbfd-e6e174be22aa-kube-api-access-tx4kr\") pod \"maas-api-revoke-sub-dc52e205-gdnrw\" (UID: \"3b0daf67-c56b-4024-bbfd-e6e174be22aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc52e205-gdnrw" Nov 02 18:57:04.891075 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:04.890918 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0daf67-c56b-4024-bbfd-e6e174be22aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc52e205-gdnrw\" (UID: \"3b0daf67-c56b-4024-bbfd-e6e174be22aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc52e205-gdnrw" Nov 02 18:57:04.891075 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:04.891024 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0daf67-c56b-4024-bbfd-e6e174be22aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc52e205-gdnrw\" (UID: \"3b0daf67-c56b-4024-bbfd-e6e174be22aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc52e205-gdnrw" Nov 02 18:57:04.992251 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:04.992193 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx4kr\" (UniqueName: \"kubernetes.io/projected/3b0daf67-c56b-4024-bbfd-e6e174be22aa-kube-api-access-tx4kr\") pod \"maas-api-revoke-sub-dc52e205-gdnrw\" (UID: \"3b0daf67-c56b-4024-bbfd-e6e174be22aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc52e205-gdnrw" Nov 02 18:57:04.992251 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:04.992254 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0daf67-c56b-4024-bbfd-e6e174be22aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc52e205-gdnrw\" (UID: \"3b0daf67-c56b-4024-bbfd-e6e174be22aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc52e205-gdnrw" Nov 02 18:57:04.992543 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:04.992279 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0daf67-c56b-4024-bbfd-e6e174be22aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc52e205-gdnrw\" (UID: \"3b0daf67-c56b-4024-bbfd-e6e174be22aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc52e205-gdnrw" Nov 02 18:57:04.992944 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:04.992924 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0daf67-c56b-4024-bbfd-e6e174be22aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc52e205-gdnrw\" (UID: \"3b0daf67-c56b-4024-bbfd-e6e174be22aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc52e205-gdnrw" Nov 02 18:57:04.994762 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:04.994739 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0daf67-c56b-4024-bbfd-e6e174be22aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc52e205-gdnrw\" (UID: \"3b0daf67-c56b-4024-bbfd-e6e174be22aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc52e205-gdnrw" Nov 02 18:57:05.000278 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:05.000252 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx4kr\" (UniqueName: \"kubernetes.io/projected/3b0daf67-c56b-4024-bbfd-e6e174be22aa-kube-api-access-tx4kr\") pod \"maas-api-revoke-sub-dc52e205-gdnrw\" (UID: \"3b0daf67-c56b-4024-bbfd-e6e174be22aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc52e205-gdnrw" Nov 02 18:57:05.181212 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:05.181176 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc52e205-gdnrw" Nov 02 18:57:05.457500 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:05.457471 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc52e205-gdnrw"] Nov 02 18:57:05.459789 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:57:05.459757 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0daf67_c56b_4024_bbfd_e6e174be22aa.slice/crio-f1247c3459b8d078b8189c502f02d0d8d1868bb5ae90ad843d2918a56126f881 WatchSource:0}: Error finding container f1247c3459b8d078b8189c502f02d0d8d1868bb5ae90ad843d2918a56126f881: Status 404 returned error can't find the container with id f1247c3459b8d078b8189c502f02d0d8d1868bb5ae90ad843d2918a56126f881 Nov 02 18:57:05.663039 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:05.663003 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc52e205-gdnrw" event={"ID":"3b0daf67-c56b-4024-bbfd-e6e174be22aa","Type":"ContainerStarted","Data":"f1247c3459b8d078b8189c502f02d0d8d1868bb5ae90ad843d2918a56126f881"} Nov 02 18:57:06.673228 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:06.673189 5601 generic.go:356] "Generic (PLEG): container finished" podID="3b0daf67-c56b-4024-bbfd-e6e174be22aa" containerID="68ad61014b19b305224bed752f52667ef518442488e20ac190fe6307faf2c45c" exitCode=0 Nov 02 18:57:06.673751 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:06.673292 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc52e205-gdnrw" event={"ID":"3b0daf67-c56b-4024-bbfd-e6e174be22aa","Type":"ContainerDied","Data":"68ad61014b19b305224bed752f52667ef518442488e20ac190fe6307faf2c45c"} Nov 02 18:57:08.033281 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:08.033255 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc52e205-gdnrw" Nov 02 18:57:08.128715 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:08.128674 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0daf67-c56b-4024-bbfd-e6e174be22aa-kube-api-access-tx4kr\" (UniqueName: \"kubernetes.io/projected/3b0daf67-c56b-4024-bbfd-e6e174be22aa-kube-api-access-tx4kr\") pod \"3b0daf67-c56b-4024-bbfd-e6e174be22aa\" (UID: \"3b0daf67-c56b-4024-bbfd-e6e174be22aa\") " Nov 02 18:57:08.128915 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:08.128794 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b0daf67-c56b-4024-bbfd-e6e174be22aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0daf67-c56b-4024-bbfd-e6e174be22aa-maas-api-openshift-ca\") pod \"3b0daf67-c56b-4024-bbfd-e6e174be22aa\" (UID: \"3b0daf67-c56b-4024-bbfd-e6e174be22aa\") " Nov 02 18:57:08.128915 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:08.128899 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b0daf67-c56b-4024-bbfd-e6e174be22aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0daf67-c56b-4024-bbfd-e6e174be22aa-maas-api-xks-ca\") pod \"3b0daf67-c56b-4024-bbfd-e6e174be22aa\" (UID: \"3b0daf67-c56b-4024-bbfd-e6e174be22aa\") " Nov 02 18:57:08.129462 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:08.129433 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0daf67-c56b-4024-bbfd-e6e174be22aa-maas-api-openshift-ca" pod "3b0daf67-c56b-4024-bbfd-e6e174be22aa" (UID: "3b0daf67-c56b-4024-bbfd-e6e174be22aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:57:08.131323 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:08.131297 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b0daf67-c56b-4024-bbfd-e6e174be22aa-maas-api-xks-ca" pod "3b0daf67-c56b-4024-bbfd-e6e174be22aa" (UID: "3b0daf67-c56b-4024-bbfd-e6e174be22aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:57:08.131431 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:08.131332 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0daf67-c56b-4024-bbfd-e6e174be22aa-kube-api-access-tx4kr" pod "3b0daf67-c56b-4024-bbfd-e6e174be22aa" (UID: "3b0daf67-c56b-4024-bbfd-e6e174be22aa"). InnerVolumeSpecName "kube-api-access-tx4kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:57:08.230323 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:08.230272 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0daf67-c56b-4024-bbfd-e6e174be22aa-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:57:08.230323 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:08.230307 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0daf67-c56b-4024-bbfd-e6e174be22aa-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:57:08.230323 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:08.230318 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx4kr\" (UniqueName: \"kubernetes.io/projected/3b0daf67-c56b-4024-bbfd-e6e174be22aa-kube-api-access-tx4kr\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:57:08.373701 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:08.373569 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-th4lt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:57:08.373701 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:08.373661 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" podUID="0659edff-3cd9-495d-bbce-c19f48452bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:57:08.701892 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:08.701858 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc52e205-gdnrw" Nov 02 18:57:08.702115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:08.701891 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc52e205-gdnrw" event={"ID":"3b0daf67-c56b-4024-bbfd-e6e174be22aa","Type":"ContainerDied","Data":"f1247c3459b8d078b8189c502f02d0d8d1868bb5ae90ad843d2918a56126f881"} Nov 02 18:57:08.702115 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:08.701969 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1247c3459b8d078b8189c502f02d0d8d1868bb5ae90ad843d2918a56126f881" Nov 02 18:57:15.057959 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.057917 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88dd5e69-wqmlx"] Nov 02 18:57:15.058369 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.058294 5601 cpu_manager.go:414] "containerMap: removing container" podUID="3b0daf67-c56b-4024-bbfd-e6e174be22aa" containerName="revoke-keys" Nov 02 18:57:15.058369 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.058306 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0daf67-c56b-4024-bbfd-e6e174be22aa" containerName="revoke-keys" Nov 02 18:57:15.058469 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.058384 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0daf67-c56b-4024-bbfd-e6e174be22aa" containerName="revoke-keys" Nov 02 18:57:15.061483 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.061464 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dd5e69-wqmlx" Nov 02 18:57:15.066950 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.066925 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:57:15.067154 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.067053 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:57:15.069423 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.069396 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88dd5e69-wqmlx"] Nov 02 18:57:15.193656 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.193584 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2456d8e2-64d1-4dac-9bae-886f52d91e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88dd5e69-wqmlx\" (UID: \"2456d8e2-64d1-4dac-9bae-886f52d91e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dd5e69-wqmlx" Nov 02 18:57:15.193852 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.193746 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2456d8e2-64d1-4dac-9bae-886f52d91e7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88dd5e69-wqmlx\" (UID: \"2456d8e2-64d1-4dac-9bae-886f52d91e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dd5e69-wqmlx" Nov 02 18:57:15.193852 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.193800 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84vb2\" (UniqueName: \"kubernetes.io/projected/2456d8e2-64d1-4dac-9bae-886f52d91e7c-kube-api-access-84vb2\") pod \"maas-api-revoke-sub-88dd5e69-wqmlx\" (UID: \"2456d8e2-64d1-4dac-9bae-886f52d91e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dd5e69-wqmlx" Nov 02 18:57:15.295176 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.295140 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2456d8e2-64d1-4dac-9bae-886f52d91e7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88dd5e69-wqmlx\" (UID: \"2456d8e2-64d1-4dac-9bae-886f52d91e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dd5e69-wqmlx" Nov 02 18:57:15.295357 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.295190 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84vb2\" (UniqueName: \"kubernetes.io/projected/2456d8e2-64d1-4dac-9bae-886f52d91e7c-kube-api-access-84vb2\") pod \"maas-api-revoke-sub-88dd5e69-wqmlx\" (UID: \"2456d8e2-64d1-4dac-9bae-886f52d91e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dd5e69-wqmlx" Nov 02 18:57:15.295357 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.295244 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2456d8e2-64d1-4dac-9bae-886f52d91e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88dd5e69-wqmlx\" (UID: \"2456d8e2-64d1-4dac-9bae-886f52d91e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dd5e69-wqmlx" Nov 02 18:57:15.295925 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.295906 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2456d8e2-64d1-4dac-9bae-886f52d91e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88dd5e69-wqmlx\" (UID: \"2456d8e2-64d1-4dac-9bae-886f52d91e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dd5e69-wqmlx" Nov 02 18:57:15.297540 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.297516 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2456d8e2-64d1-4dac-9bae-886f52d91e7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88dd5e69-wqmlx\" (UID: \"2456d8e2-64d1-4dac-9bae-886f52d91e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dd5e69-wqmlx" Nov 02 18:57:15.303226 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.303194 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84vb2\" (UniqueName: \"kubernetes.io/projected/2456d8e2-64d1-4dac-9bae-886f52d91e7c-kube-api-access-84vb2\") pod \"maas-api-revoke-sub-88dd5e69-wqmlx\" (UID: \"2456d8e2-64d1-4dac-9bae-886f52d91e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dd5e69-wqmlx" Nov 02 18:57:15.377934 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.377857 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dd5e69-wqmlx" Nov 02 18:57:15.651742 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.651716 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88dd5e69-wqmlx"] Nov 02 18:57:15.653396 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:57:15.653368 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2456d8e2_64d1_4dac_9bae_886f52d91e7c.slice/crio-57941ca0285f17536ddef9dfd3c0a6072341d741f8c34c77f3f065c3b2102da8 WatchSource:0}: Error finding container 57941ca0285f17536ddef9dfd3c0a6072341d741f8c34c77f3f065c3b2102da8: Status 404 returned error can't find the container with id 57941ca0285f17536ddef9dfd3c0a6072341d741f8c34c77f3f065c3b2102da8 Nov 02 18:57:15.740549 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:15.740513 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dd5e69-wqmlx" event={"ID":"2456d8e2-64d1-4dac-9bae-886f52d91e7c","Type":"ContainerStarted","Data":"57941ca0285f17536ddef9dfd3c0a6072341d741f8c34c77f3f065c3b2102da8"} Nov 02 18:57:16.749398 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:16.749365 5601 generic.go:356] "Generic (PLEG): container finished" podID="2456d8e2-64d1-4dac-9bae-886f52d91e7c" containerID="ff558cd1670b69a2c2bf5f9906639e98840e3c9bf6019a597dfe685672666a75" exitCode=0 Nov 02 18:57:16.749960 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:16.749436 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dd5e69-wqmlx" event={"ID":"2456d8e2-64d1-4dac-9bae-886f52d91e7c","Type":"ContainerDied","Data":"ff558cd1670b69a2c2bf5f9906639e98840e3c9bf6019a597dfe685672666a75"} Nov 02 18:57:18.129609 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:18.129579 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dd5e69-wqmlx" Nov 02 18:57:18.226303 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:18.226268 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2456d8e2-64d1-4dac-9bae-886f52d91e7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2456d8e2-64d1-4dac-9bae-886f52d91e7c-maas-api-xks-ca\") pod \"2456d8e2-64d1-4dac-9bae-886f52d91e7c\" (UID: \"2456d8e2-64d1-4dac-9bae-886f52d91e7c\") " Nov 02 18:57:18.226507 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:18.226320 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2456d8e2-64d1-4dac-9bae-886f52d91e7c-kube-api-access-84vb2\" (UniqueName: \"kubernetes.io/projected/2456d8e2-64d1-4dac-9bae-886f52d91e7c-kube-api-access-84vb2\") pod \"2456d8e2-64d1-4dac-9bae-886f52d91e7c\" (UID: \"2456d8e2-64d1-4dac-9bae-886f52d91e7c\") " Nov 02 18:57:18.226507 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:18.226354 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2456d8e2-64d1-4dac-9bae-886f52d91e7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2456d8e2-64d1-4dac-9bae-886f52d91e7c-maas-api-openshift-ca\") pod \"2456d8e2-64d1-4dac-9bae-886f52d91e7c\" (UID: \"2456d8e2-64d1-4dac-9bae-886f52d91e7c\") " Nov 02 18:57:18.226830 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:18.226807 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2456d8e2-64d1-4dac-9bae-886f52d91e7c-maas-api-openshift-ca" pod "2456d8e2-64d1-4dac-9bae-886f52d91e7c" (UID: "2456d8e2-64d1-4dac-9bae-886f52d91e7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:57:18.228346 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:18.228318 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2456d8e2-64d1-4dac-9bae-886f52d91e7c-maas-api-xks-ca" pod "2456d8e2-64d1-4dac-9bae-886f52d91e7c" (UID: "2456d8e2-64d1-4dac-9bae-886f52d91e7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:57:18.228521 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:18.228504 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2456d8e2-64d1-4dac-9bae-886f52d91e7c-kube-api-access-84vb2" pod "2456d8e2-64d1-4dac-9bae-886f52d91e7c" (UID: "2456d8e2-64d1-4dac-9bae-886f52d91e7c"). InnerVolumeSpecName "kube-api-access-84vb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:57:18.327731 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:18.327689 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2456d8e2-64d1-4dac-9bae-886f52d91e7c-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:57:18.327731 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:18.327727 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84vb2\" (UniqueName: \"kubernetes.io/projected/2456d8e2-64d1-4dac-9bae-886f52d91e7c-kube-api-access-84vb2\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:57:18.327949 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:18.327741 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2456d8e2-64d1-4dac-9bae-886f52d91e7c-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:57:18.773993 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:18.773958 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dd5e69-wqmlx" Nov 02 18:57:18.773993 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:18.773973 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dd5e69-wqmlx" event={"ID":"2456d8e2-64d1-4dac-9bae-886f52d91e7c","Type":"ContainerDied","Data":"57941ca0285f17536ddef9dfd3c0a6072341d741f8c34c77f3f065c3b2102da8"} Nov 02 18:57:18.774207 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:18.774016 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57941ca0285f17536ddef9dfd3c0a6072341d741f8c34c77f3f065c3b2102da8" Nov 02 18:57:25.360733 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:25.360698 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97"] Nov 02 18:57:25.361148 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:25.361100 5601 cpu_manager.go:414] "containerMap: removing container" podUID="2456d8e2-64d1-4dac-9bae-886f52d91e7c" containerName="revoke-keys" Nov 02 18:57:25.361148 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:25.361111 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2456d8e2-64d1-4dac-9bae-886f52d91e7c" containerName="revoke-keys" Nov 02 18:57:25.361224 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:25.361192 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="2456d8e2-64d1-4dac-9bae-886f52d91e7c" containerName="revoke-keys" Nov 02 18:57:25.364574 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:25.364553 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97" Nov 02 18:57:25.369905 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:25.369879 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:57:25.370058 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:25.369951 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:57:25.373676 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:25.373146 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97"] Nov 02 18:57:25.497665 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:25.497609 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfgbr\" (UniqueName: \"kubernetes.io/projected/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-kube-api-access-xfgbr\") pod \"maas-api-revoke-sub-e2c7204c-xwc97\" (UID: \"7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97" Nov 02 18:57:25.497844 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:25.497686 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c7204c-xwc97\" (UID: \"7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97" Nov 02 18:57:25.497844 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:25.497821 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c7204c-xwc97\" (UID: \"7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97" Nov 02 18:57:25.598300 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:25.598256 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfgbr\" (UniqueName: \"kubernetes.io/projected/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-kube-api-access-xfgbr\") pod \"maas-api-revoke-sub-e2c7204c-xwc97\" (UID: \"7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97" Nov 02 18:57:25.598514 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:25.598311 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c7204c-xwc97\" (UID: \"7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97" Nov 02 18:57:25.598514 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:25.598395 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c7204c-xwc97\" (UID: \"7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97" Nov 02 18:57:25.599054 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:25.599031 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c7204c-xwc97\" (UID: \"7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97" Nov 02 18:57:25.600942 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:25.600914 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c7204c-xwc97\" (UID: \"7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97" Nov 02 18:57:25.606344 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:25.606320 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfgbr\" (UniqueName: \"kubernetes.io/projected/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-kube-api-access-xfgbr\") pod \"maas-api-revoke-sub-e2c7204c-xwc97\" (UID: \"7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97" Nov 02 18:57:25.680385 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:25.680342 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97" Nov 02 18:57:25.955764 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:25.955737 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97"] Nov 02 18:57:25.957880 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:57:25.957845 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa76e48_d4b6_4c20_8adf_9ea0296cf7e1.slice/crio-ab4b3b02f20e53babe59d478d540ca8d0c5dfd15cb395d836dbf2b92c6f4592b WatchSource:0}: Error finding container ab4b3b02f20e53babe59d478d540ca8d0c5dfd15cb395d836dbf2b92c6f4592b: Status 404 returned error can't find the container with id ab4b3b02f20e53babe59d478d540ca8d0c5dfd15cb395d836dbf2b92c6f4592b Nov 02 18:57:26.827248 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:26.827200 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97" event={"ID":"7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1","Type":"ContainerStarted","Data":"7af527e75b77e1af7aabbff48469be49b599f7fba9cbeea15a82918060dbf7c1"} Nov 02 18:57:26.827827 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:26.827275 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97" event={"ID":"7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1","Type":"ContainerStarted","Data":"ab4b3b02f20e53babe59d478d540ca8d0c5dfd15cb395d836dbf2b92c6f4592b"} Nov 02 18:57:26.845670 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:26.845594 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97" podStartSLOduration=1.845574646 podStartE2EDuration="1.845574646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:57:26.842221341 +0000 UTC m=+5011.533275354" watchObservedRunningTime="2026-11-02 18:57:26.845574646 +0000 UTC m=+5011.536628659" Nov 02 18:57:27.841154 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:27.841104 5601 generic.go:356] "Generic (PLEG): container finished" podID="7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1" containerID="7af527e75b77e1af7aabbff48469be49b599f7fba9cbeea15a82918060dbf7c1" exitCode=0 Nov 02 18:57:27.845001 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:27.841982 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97" event={"ID":"7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1","Type":"ContainerDied","Data":"7af527e75b77e1af7aabbff48469be49b599f7fba9cbeea15a82918060dbf7c1"} Nov 02 18:57:29.000970 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:29.000943 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97" Nov 02 18:57:29.030857 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:29.030829 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-maas-api-openshift-ca\") pod \"7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1\" (UID: \"7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1\") " Nov 02 18:57:29.031008 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:29.030935 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-kube-api-access-xfgbr\" (UniqueName: \"kubernetes.io/projected/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-kube-api-access-xfgbr\") pod \"7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1\" (UID: \"7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1\") " Nov 02 18:57:29.031008 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:29.030997 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-maas-api-xks-ca\") pod \"7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1\" (UID: \"7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1\") " Nov 02 18:57:29.031240 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:29.031215 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-maas-api-openshift-ca" pod "7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1" (UID: "7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:57:29.031320 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:29.031292 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:57:29.033023 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:29.032993 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-kube-api-access-xfgbr" pod "7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1" (UID: "7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1"). InnerVolumeSpecName "kube-api-access-xfgbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:57:29.033120 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:29.033019 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-maas-api-xks-ca" pod "7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1" (UID: "7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:57:29.132234 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:29.132145 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfgbr\" (UniqueName: \"kubernetes.io/projected/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-kube-api-access-xfgbr\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:57:29.132234 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:29.132182 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:57:29.857179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:29.857104 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97" Nov 02 18:57:29.857179 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:29.857134 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c7204c-xwc97" event={"ID":"7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1","Type":"ContainerDied","Data":"ab4b3b02f20e53babe59d478d540ca8d0c5dfd15cb395d836dbf2b92c6f4592b"} Nov 02 18:57:29.857379 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:29.857210 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab4b3b02f20e53babe59d478d540ca8d0c5dfd15cb395d836dbf2b92c6f4592b" Nov 02 18:57:34.484250 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:57:34.484210 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:57:35.562087 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:35.562045 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg"] Nov 02 18:57:35.562663 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:35.562643 5601 cpu_manager.go:414] "containerMap: removing container" podUID="7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1" containerName="revoke-keys" Nov 02 18:57:35.562761 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:35.562665 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1" containerName="revoke-keys" Nov 02 18:57:35.562831 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:35.562822 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1" containerName="revoke-keys" Nov 02 18:57:35.566034 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:35.566013 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg" Nov 02 18:57:35.570901 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:35.570875 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:57:35.571067 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:35.570976 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:57:35.573793 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:35.573766 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg"] Nov 02 18:57:35.694015 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:35.693976 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9de8901-jczzg\" (UID: \"31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg" Nov 02 18:57:35.694015 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:35.694016 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9de8901-jczzg\" (UID: \"31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg" Nov 02 18:57:35.694258 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:35.694051 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pvkd\" (UniqueName: \"kubernetes.io/projected/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-kube-api-access-6pvkd\") pod \"maas-api-revoke-sub-a9de8901-jczzg\" (UID: \"31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg" Nov 02 18:57:35.794820 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:35.794772 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9de8901-jczzg\" (UID: \"31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg" Nov 02 18:57:35.794820 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:35.794822 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9de8901-jczzg\" (UID: \"31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg" Nov 02 18:57:35.795088 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:35.794851 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pvkd\" (UniqueName: \"kubernetes.io/projected/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-kube-api-access-6pvkd\") pod \"maas-api-revoke-sub-a9de8901-jczzg\" (UID: \"31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg" Nov 02 18:57:35.795558 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:35.795535 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9de8901-jczzg\" (UID: \"31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg" Nov 02 18:57:35.797230 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:35.797210 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9de8901-jczzg\" (UID: \"31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg" Nov 02 18:57:35.802872 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:35.802847 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pvkd\" (UniqueName: \"kubernetes.io/projected/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-kube-api-access-6pvkd\") pod \"maas-api-revoke-sub-a9de8901-jczzg\" (UID: \"31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg" Nov 02 18:57:35.880916 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:35.880834 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg" Nov 02 18:57:36.155182 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:36.155143 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg"] Nov 02 18:57:36.156682 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:57:36.156655 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e5a6f4_93a3_4fb5_8ca3_cf5137cf4eb2.slice/crio-ed62678a62597d1b81c8557158b456a039986bc308418923d5b5f472fb6c32d1 WatchSource:0}: Error finding container ed62678a62597d1b81c8557158b456a039986bc308418923d5b5f472fb6c32d1: Status 404 returned error can't find the container with id ed62678a62597d1b81c8557158b456a039986bc308418923d5b5f472fb6c32d1 Nov 02 18:57:36.904342 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:36.904245 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg" event={"ID":"31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2","Type":"ContainerStarted","Data":"0ab252f5517b9d3440df7779815f8dfa765fcdeac39e2c50f40ecfe49365c6f0"} Nov 02 18:57:36.904990 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:36.904959 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg" event={"ID":"31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2","Type":"ContainerStarted","Data":"ed62678a62597d1b81c8557158b456a039986bc308418923d5b5f472fb6c32d1"} Nov 02 18:57:36.934216 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:36.934158 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg" podStartSLOduration=1.934141576 podStartE2EDuration="1.934141576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:57:36.933444949 +0000 UTC m=+5021.624498962" watchObservedRunningTime="2026-11-02 18:57:36.934141576 +0000 UTC m=+5021.625195591" Nov 02 18:57:37.915078 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:37.915039 5601 generic.go:356] "Generic (PLEG): container finished" podID="31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2" containerID="0ab252f5517b9d3440df7779815f8dfa765fcdeac39e2c50f40ecfe49365c6f0" exitCode=0 Nov 02 18:57:37.915508 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:37.915122 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg" event={"ID":"31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2","Type":"ContainerDied","Data":"0ab252f5517b9d3440df7779815f8dfa765fcdeac39e2c50f40ecfe49365c6f0"} Nov 02 18:57:38.413790 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:38.413747 5601 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-th4lt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:57:38.413994 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:38.413820 5601 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-th4lt" podUID="0659edff-3cd9-495d-bbce-c19f48452bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:57:39.142104 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:39.142069 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg" Nov 02 18:57:39.231780 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:39.231706 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-maas-api-xks-ca\") pod \"31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2\" (UID: \"31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2\") " Nov 02 18:57:39.231928 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:39.231781 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-kube-api-access-6pvkd\" (UniqueName: \"kubernetes.io/projected/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-kube-api-access-6pvkd\") pod \"31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2\" (UID: \"31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2\") " Nov 02 18:57:39.231928 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:39.231842 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-maas-api-openshift-ca\") pod \"31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2\" (UID: \"31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2\") " Nov 02 18:57:39.232229 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:39.232205 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-maas-api-openshift-ca" pod "31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2" (UID: "31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:57:39.233925 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:39.233897 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-maas-api-xks-ca" pod "31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2" (UID: "31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:57:39.234033 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:39.233942 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-kube-api-access-6pvkd" pod "31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2" (UID: "31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2"). InnerVolumeSpecName "kube-api-access-6pvkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:57:39.333037 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:39.333003 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:57:39.333037 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:39.333034 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pvkd\" (UniqueName: \"kubernetes.io/projected/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-kube-api-access-6pvkd\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:57:39.333215 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:39.333043 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:57:39.929121 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:39.929086 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg" Nov 02 18:57:39.929121 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:39.929102 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9de8901-jczzg" event={"ID":"31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2","Type":"ContainerDied","Data":"ed62678a62597d1b81c8557158b456a039986bc308418923d5b5f472fb6c32d1"} Nov 02 18:57:39.929335 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:39.929145 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed62678a62597d1b81c8557158b456a039986bc308418923d5b5f472fb6c32d1" Nov 02 18:57:45.764187 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:45.764151 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk"] Nov 02 18:57:45.764586 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:45.764560 5601 cpu_manager.go:414] "containerMap: removing container" podUID="31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2" containerName="revoke-keys" Nov 02 18:57:45.764586 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:45.764571 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2" containerName="revoke-keys" Nov 02 18:57:45.764705 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:45.764685 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2" containerName="revoke-keys" Nov 02 18:57:45.767962 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:45.767937 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk" Nov 02 18:57:45.773781 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:45.773753 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\"" Nov 02 18:57:45.776507 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:45.776483 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:57:45.778927 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:45.778899 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk"] Nov 02 18:57:45.894926 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:45.894891 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6xr4\" (UniqueName: \"kubernetes.io/projected/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-kube-api-access-x6xr4\") pod \"maas-api-revoke-sub-c4244650-px5mk\" (UID: \"1d808173-ae52-4ac6-b7f4-3e09f3d7d3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk" Nov 02 18:57:45.895074 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:45.894993 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4244650-px5mk\" (UID: \"1d808173-ae52-4ac6-b7f4-3e09f3d7d3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk" Nov 02 18:57:45.895074 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:45.895038 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4244650-px5mk\" (UID: \"1d808173-ae52-4ac6-b7f4-3e09f3d7d3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk" Nov 02 18:57:45.995865 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:45.995830 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4244650-px5mk\" (UID: \"1d808173-ae52-4ac6-b7f4-3e09f3d7d3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk" Nov 02 18:57:45.996039 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:45.995879 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4244650-px5mk\" (UID: \"1d808173-ae52-4ac6-b7f4-3e09f3d7d3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk" Nov 02 18:57:45.996039 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:45.995940 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6xr4\" (UniqueName: \"kubernetes.io/projected/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-kube-api-access-x6xr4\") pod \"maas-api-revoke-sub-c4244650-px5mk\" (UID: \"1d808173-ae52-4ac6-b7f4-3e09f3d7d3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk" Nov 02 18:57:45.996501 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:45.996470 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4244650-px5mk\" (UID: \"1d808173-ae52-4ac6-b7f4-3e09f3d7d3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk" Nov 02 18:57:45.998339 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:45.998309 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4244650-px5mk\" (UID: \"1d808173-ae52-4ac6-b7f4-3e09f3d7d3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk" Nov 02 18:57:46.004261 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:46.004239 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6xr4\" (UniqueName: \"kubernetes.io/projected/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-kube-api-access-x6xr4\") pod \"maas-api-revoke-sub-c4244650-px5mk\" (UID: \"1d808173-ae52-4ac6-b7f4-3e09f3d7d3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk" Nov 02 18:57:46.083335 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:46.083261 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk" Nov 02 18:57:46.360626 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:46.360588 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk"] Nov 02 18:57:46.362041 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:57:46.362013 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d808173_ae52_4ac6_b7f4_3e09f3d7d3da.slice/crio-f7e84044576e76aef60c8f0f5621c39a35fc8373f89d30ee143df630c245ea5e WatchSource:0}: Error finding container f7e84044576e76aef60c8f0f5621c39a35fc8373f89d30ee143df630c245ea5e: Status 404 returned error can't find the container with id f7e84044576e76aef60c8f0f5621c39a35fc8373f89d30ee143df630c245ea5e Nov 02 18:57:47.005088 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:47.005033 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk" event={"ID":"1d808173-ae52-4ac6-b7f4-3e09f3d7d3da","Type":"ContainerStarted","Data":"5d06783883fb3540161487c1876890abcfa7678dae1dcd27f062924de64aac29"} Nov 02 18:57:47.005602 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:47.005108 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk" event={"ID":"1d808173-ae52-4ac6-b7f4-3e09f3d7d3da","Type":"ContainerStarted","Data":"f7e84044576e76aef60c8f0f5621c39a35fc8373f89d30ee143df630c245ea5e"} Nov 02 18:57:47.024279 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:47.024226 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk" podStartSLOduration=2.024213711 podStartE2EDuration="2.024213711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:57:47.022454373 +0000 UTC m=+5031.713508386" watchObservedRunningTime="2026-11-02 18:57:47.024213711 +0000 UTC m=+5031.715267716" Nov 02 18:57:48.015920 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:48.015888 5601 generic.go:356] "Generic (PLEG): container finished" podID="1d808173-ae52-4ac6-b7f4-3e09f3d7d3da" containerID="5d06783883fb3540161487c1876890abcfa7678dae1dcd27f062924de64aac29" exitCode=0 Nov 02 18:57:48.016447 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:48.015989 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk" event={"ID":"1d808173-ae52-4ac6-b7f4-3e09f3d7d3da","Type":"ContainerDied","Data":"5d06783883fb3540161487c1876890abcfa7678dae1dcd27f062924de64aac29"} Nov 02 18:57:49.153803 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:49.153779 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk" Nov 02 18:57:49.237251 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:49.237211 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-kube-api-access-x6xr4\" (UniqueName: \"kubernetes.io/projected/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-kube-api-access-x6xr4\") pod \"1d808173-ae52-4ac6-b7f4-3e09f3d7d3da\" (UID: \"1d808173-ae52-4ac6-b7f4-3e09f3d7d3da\") " Nov 02 18:57:49.237456 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:49.237281 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-maas-api-openshift-ca\") pod \"1d808173-ae52-4ac6-b7f4-3e09f3d7d3da\" (UID: \"1d808173-ae52-4ac6-b7f4-3e09f3d7d3da\") " Nov 02 18:57:49.237456 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:49.237308 5601 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-maas-api-xks-ca\") pod \"1d808173-ae52-4ac6-b7f4-3e09f3d7d3da\" (UID: \"1d808173-ae52-4ac6-b7f4-3e09f3d7d3da\") " Nov 02 18:57:49.237688 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:49.237645 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-maas-api-openshift-ca" pod "1d808173-ae52-4ac6-b7f4-3e09f3d7d3da" (UID: "1d808173-ae52-4ac6-b7f4-3e09f3d7d3da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:57:49.239479 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:49.239456 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-maas-api-xks-ca" pod "1d808173-ae52-4ac6-b7f4-3e09f3d7d3da" (UID: "1d808173-ae52-4ac6-b7f4-3e09f3d7d3da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:57:49.239546 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:49.239475 5601 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-kube-api-access-x6xr4" pod "1d808173-ae52-4ac6-b7f4-3e09f3d7d3da" (UID: "1d808173-ae52-4ac6-b7f4-3e09f3d7d3da"). InnerVolumeSpecName "kube-api-access-x6xr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:57:49.338963 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:49.338882 5601 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6xr4\" (UniqueName: \"kubernetes.io/projected/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-kube-api-access-x6xr4\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:57:49.338963 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:49.338913 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-maas-api-openshift-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:57:49.338963 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:49.338925 5601 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d808173-ae52-4ac6-b7f4-3e09f3d7d3da-maas-api-xks-ca\") on node \"ip-10-0-141-169.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:57:50.029510 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:50.029478 5601 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk" Nov 02 18:57:50.029709 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:50.029516 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4244650-px5mk" event={"ID":"1d808173-ae52-4ac6-b7f4-3e09f3d7d3da","Type":"ContainerDied","Data":"f7e84044576e76aef60c8f0f5621c39a35fc8373f89d30ee143df630c245ea5e"} Nov 02 18:57:50.029709 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:57:50.029571 5601 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7e84044576e76aef60c8f0f5621c39a35fc8373f89d30ee143df630c245ea5e" Nov 02 18:58:04.483834 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:58:04.483797 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:58:20.564471 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:20.564441 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-t5dg5_aad4749c-18d8-4003-8576-211672ff42e5/manager/0.log" Nov 02 18:58:20.967123 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:20.967090 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-hsptq_3aa1eac9-b956-4608-8c5a-343b37a69d70/manager/0.log" Nov 02 18:58:23.426654 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:23.426607 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0245bb0d-xtfvr_479f2e22-3f3e-40f8-8fb6-3b98d34373c3/revoke-keys/0.log" Nov 02 18:58:23.610347 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:23.610318 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1b3208e8-h9v54_d3a14d7d-9aba-4dd2-af15-5a5853c5cf04/revoke-keys/0.log" Nov 02 18:58:23.800271 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:23.800179 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-20fbe5c3-dn5vn_4c23979e-84bc-45f9-a989-13d2989bd8d6/revoke-keys/0.log" Nov 02 18:58:23.991398 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:23.991363 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-418bf626-n8lpd_9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6/revoke-keys/0.log" Nov 02 18:58:24.186186 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:24.186151 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-495c507e-qzrt9_b5c49fcb-f23e-4366-b5eb-a0e26899e33d/revoke-keys/0.log" Nov 02 18:58:24.372164 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:24.372130 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-88c5abea-vl9v2_02762677-23a1-4a96-b27f-036a2baf746c/revoke-keys/0.log" Nov 02 18:58:24.559199 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:24.559123 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-88dd5e69-wqmlx_2456d8e2-64d1-4dac-9bae-886f52d91e7c/revoke-keys/0.log" Nov 02 18:58:24.741385 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:24.741356 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-97ab0ecc-462bv_0a3483e4-4266-4bfd-a01a-a6cb0aed3089/revoke-keys/0.log" Nov 02 18:58:24.925880 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:24.925855 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9a524851-75m25_e037e2db-712d-41a3-bff7-1bae2fc63897/revoke-keys/0.log" Nov 02 18:58:25.105399 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:25.105369 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a9de8901-jczzg_31e5a6f4-93a3-4fb5-8ca3-cf5137cf4eb2/revoke-keys/0.log" Nov 02 18:58:25.298342 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:25.298268 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c4244650-px5mk_1d808173-ae52-4ac6-b7f4-3e09f3d7d3da/revoke-keys/0.log" Nov 02 18:58:25.482820 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:25.482781 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d6e66f0f-zcsqg_bf94ee92-6bf4-449f-b97c-80ee9cb0c41b/revoke-keys/0.log" Nov 02 18:58:25.665311 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:25.665230 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d762b3d1-297h5_cabbd5f1-2cc7-4be8-ace4-de85ca73bd0c/revoke-keys/0.log" Nov 02 18:58:25.853874 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:25.853846 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-dc52e205-gdnrw_3b0daf67-c56b-4024-bbfd-e6e174be22aa/revoke-keys/0.log" Nov 02 18:58:26.041770 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:26.041737 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e2c7204c-xwc97_7fa76e48-d4b6-4c20-8adf-9ea0296cf7e1/revoke-keys/0.log" Nov 02 18:58:26.212860 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:26.212833 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e73b344f-vz6dt_750a8b45-b55f-45fd-8f3b-1858245a4e08/revoke-keys/0.log" Nov 02 18:58:26.390022 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:26.389940 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f1700323-vt2dd_e9c1e0b7-7e96-4e49-a386-3407ad293494/revoke-keys/0.log" Nov 02 18:58:26.570490 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:26.570461 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f25c9f82-x7wdz_4e715639-a159-4bce-8e35-812b50f35ae1/revoke-keys/0.log" Nov 02 18:58:26.740966 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:26.740942 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f9d0f7b3-5gxqb_86b7d192-19fa-4e37-a839-dc373a1774b7/revoke-keys/0.log" Nov 02 18:58:26.917668 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:26.917639 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-gk2ps_0c86a124-ed3f-4592-a31a-09ad586482d1/postgres/0.log" Nov 02 18:58:31.247988 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:31.247950 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-dl596_f0e27933-7691-4793-922c-4e83bba3914e/discovery/0.log" Nov 02 18:58:31.437393 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:31.437352 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cf9f7b655-m7r89_9d1ee04f-2201-4da0-88ff-f06683fc6006/kube-auth-proxy/0.log" Nov 02 18:58:32.202680 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:32.202640 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-jv8f4_c32fa6f3-9665-4b3b-b093-22435190829b/istio-proxy/0.log" Nov 02 18:58:32.791351 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:32.791300 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-th4lt_0659edff-3cd9-495d-bbce-c19f48452bf4/istio-proxy/0.log" Nov 02 18:58:32.992424 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:32.992396 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-wtks5_63f70366-70f1-475a-8e45-eb6da6b4828e/istio-proxy/1.log" Nov 02 18:58:34.483879 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:58:34.483843 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:58:34.949415 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:34.949384 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-wtks5_63f70366-70f1-475a-8e45-eb6da6b4828e/istio-proxy/0.log" Nov 02 18:58:42.023765 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:42.023722 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd"] Nov 02 18:58:42.025963 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:42.025938 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1700323-vt2dd"] Nov 02 18:58:43.424692 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:43.424655 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c1e0b7-7e96-4e49-a386-3407ad293494" path="/var/lib/kubelet/pods/e9c1e0b7-7e96-4e49-a386-3407ad293494/volumes" Nov 02 18:58:50.753236 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:50.753195 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f4xwb_d29fb0ac-cd81-4d4c-bba7-42f9c0b686f7/global-pull-secret-syncer/0.log" Nov 02 18:58:51.034835 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:51.034740 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cjnbh_c00d16e7-b92c-46c7-a68f-797274bbef93/konnectivity-agent/0.log" Nov 02 18:58:51.143630 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:51.143597 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-169.us-east-2.compute.internal_f82e5ebcf2f8d3b5e95e3af83d2365db/haproxy/0.log" Nov 02 18:58:52.031177 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:52.031137 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd"] Nov 02 18:58:52.036797 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:52.036769 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418bf626-n8lpd"] Nov 02 18:58:53.431331 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:58:53.431295 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6" path="/var/lib/kubelet/pods/9f5a0e1f-4b06-4426-bf1a-abdbee7eb9e6/volumes" Nov 02 18:59:00.578687 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:00.578655 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-zk477_065a4c3d-712b-4b65-b18f-0428e7a93672/kube-state-metrics/0.log" Nov 02 18:59:00.597935 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:00.597910 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-zk477_065a4c3d-712b-4b65-b18f-0428e7a93672/kube-rbac-proxy-main/0.log" Nov 02 18:59:00.625078 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:00.625049 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-zk477_065a4c3d-712b-4b65-b18f-0428e7a93672/kube-rbac-proxy-self/0.log" Nov 02 18:59:00.654993 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:00.654966 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-dcf4d45cc-49kqs_c22db86a-ab8d-4a07-b3b2-6efdc24f8dc8/metrics-server/0.log" Nov 02 18:59:00.696873 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:00.696847 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-4c892_43f970d6-97ec-4e75-820b-aceefb7c37f5/monitoring-plugin/0.log" Nov 02 18:59:00.846675 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:00.846560 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8dtcc_0a0753ab-8636-4ce9-9b3e-d89516b73c1c/node-exporter/0.log" Nov 02 18:59:00.872815 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:00.872786 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8dtcc_0a0753ab-8636-4ce9-9b3e-d89516b73c1c/kube-rbac-proxy/0.log" Nov 02 18:59:00.896939 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:00.896911 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8dtcc_0a0753ab-8636-4ce9-9b3e-d89516b73c1c/init-textfile/0.log" Nov 02 18:59:01.007999 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.007968 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-9vgph_7ae7bace-4c4c-4f60-9863-5b2c29b54b76/kube-rbac-proxy-main/0.log" Nov 02 18:59:01.037664 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.037610 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-9vgph_7ae7bace-4c4c-4f60-9863-5b2c29b54b76/kube-rbac-proxy-self/0.log" Nov 02 18:59:01.061913 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.061886 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-9vgph_7ae7bace-4c4c-4f60-9863-5b2c29b54b76/openshift-state-metrics/0.log" Nov 02 18:59:01.107483 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.107405 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45570963-375c-49e4-bda8-01a095e015f7/prometheus/0.log" Nov 02 18:59:01.130592 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.130563 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45570963-375c-49e4-bda8-01a095e015f7/config-reloader/0.log" Nov 02 18:59:01.153553 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.153513 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45570963-375c-49e4-bda8-01a095e015f7/thanos-sidecar/0.log" Nov 02 18:59:01.176888 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.176842 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45570963-375c-49e4-bda8-01a095e015f7/kube-rbac-proxy-web/0.log" Nov 02 18:59:01.201273 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.201252 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45570963-375c-49e4-bda8-01a095e015f7/kube-rbac-proxy/0.log" Nov 02 18:59:01.225973 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.225951 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45570963-375c-49e4-bda8-01a095e015f7/kube-rbac-proxy-thanos/0.log" Nov 02 18:59:01.249003 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.248979 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45570963-375c-49e4-bda8-01a095e015f7/init-config-reloader/0.log" Nov 02 18:59:01.473546 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.473515 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f98f548cc-srzfq_7ecf8c2f-987e-4441-a30d-2e75f3307aae/thanos-query/0.log" Nov 02 18:59:01.497985 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.497954 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f98f548cc-srzfq_7ecf8c2f-987e-4441-a30d-2e75f3307aae/kube-rbac-proxy-web/0.log" Nov 02 18:59:01.520877 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.520841 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f98f548cc-srzfq_7ecf8c2f-987e-4441-a30d-2e75f3307aae/kube-rbac-proxy/0.log" Nov 02 18:59:01.544977 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.544946 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f98f548cc-srzfq_7ecf8c2f-987e-4441-a30d-2e75f3307aae/prom-label-proxy/0.log" Nov 02 18:59:01.568591 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.568559 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f98f548cc-srzfq_7ecf8c2f-987e-4441-a30d-2e75f3307aae/kube-rbac-proxy-rules/0.log" Nov 02 18:59:01.592230 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.592193 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f98f548cc-srzfq_7ecf8c2f-987e-4441-a30d-2e75f3307aae/kube-rbac-proxy-metrics/0.log" Nov 02 18:59:01.881045 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.880959 5601 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6"] Nov 02 18:59:01.881547 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.881522 5601 cpu_manager.go:414] "containerMap: removing container" podUID="1d808173-ae52-4ac6-b7f4-3e09f3d7d3da" containerName="revoke-keys" Nov 02 18:59:01.881547 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.881546 5601 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d808173-ae52-4ac6-b7f4-3e09f3d7d3da" containerName="revoke-keys" Nov 02 18:59:01.881775 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.881676 5601 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d808173-ae52-4ac6-b7f4-3e09f3d7d3da" containerName="revoke-keys" Nov 02 18:59:01.886296 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.886275 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" Nov 02 18:59:01.894596 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.894568 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6"] Nov 02 18:59:01.984081 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.984044 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nc6cx\"/\"kube-root-ca.crt\"" Nov 02 18:59:01.984277 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.984106 5601 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nc6cx\"/\"openshift-service-ca.crt\"" Nov 02 18:59:01.984277 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:01.984156 5601 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nc6cx\"/\"default-dockercfg-qrpjr\"" Nov 02 18:59:02.016716 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:02.016676 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/289b452d-7816-44a9-a028-d5d5844060c6-sys\") pod \"perf-node-gather-daemonset-5lpz6\" (UID: \"289b452d-7816-44a9-a028-d5d5844060c6\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" Nov 02 18:59:02.016836 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:02.016745 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hn6z\" (UniqueName: \"kubernetes.io/projected/289b452d-7816-44a9-a028-d5d5844060c6-kube-api-access-4hn6z\") pod \"perf-node-gather-daemonset-5lpz6\" (UID: \"289b452d-7816-44a9-a028-d5d5844060c6\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" Nov 02 18:59:02.016836 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:02.016827 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/289b452d-7816-44a9-a028-d5d5844060c6-podres\") pod \"perf-node-gather-daemonset-5lpz6\" (UID: \"289b452d-7816-44a9-a028-d5d5844060c6\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" Nov 02 18:59:02.016924 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:02.016873 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/289b452d-7816-44a9-a028-d5d5844060c6-proc\") pod \"perf-node-gather-daemonset-5lpz6\" (UID: \"289b452d-7816-44a9-a028-d5d5844060c6\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" Nov 02 18:59:02.016924 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:02.016900 5601 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/289b452d-7816-44a9-a028-d5d5844060c6-lib-modules\") pod \"perf-node-gather-daemonset-5lpz6\" (UID: \"289b452d-7816-44a9-a028-d5d5844060c6\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" Nov 02 18:59:02.117517 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:02.117477 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/289b452d-7816-44a9-a028-d5d5844060c6-proc\") pod \"perf-node-gather-daemonset-5lpz6\" (UID: \"289b452d-7816-44a9-a028-d5d5844060c6\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" Nov 02 18:59:02.117517 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:02.117522 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/289b452d-7816-44a9-a028-d5d5844060c6-lib-modules\") pod \"perf-node-gather-daemonset-5lpz6\" (UID: \"289b452d-7816-44a9-a028-d5d5844060c6\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" Nov 02 18:59:02.117774 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:02.117563 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/289b452d-7816-44a9-a028-d5d5844060c6-sys\") pod \"perf-node-gather-daemonset-5lpz6\" (UID: \"289b452d-7816-44a9-a028-d5d5844060c6\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" Nov 02 18:59:02.117774 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:02.117601 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hn6z\" (UniqueName: \"kubernetes.io/projected/289b452d-7816-44a9-a028-d5d5844060c6-kube-api-access-4hn6z\") pod \"perf-node-gather-daemonset-5lpz6\" (UID: \"289b452d-7816-44a9-a028-d5d5844060c6\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" Nov 02 18:59:02.117774 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:02.117626 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/289b452d-7816-44a9-a028-d5d5844060c6-proc\") pod \"perf-node-gather-daemonset-5lpz6\" (UID: \"289b452d-7816-44a9-a028-d5d5844060c6\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" Nov 02 18:59:02.117774 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:02.117692 5601 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/289b452d-7816-44a9-a028-d5d5844060c6-podres\") pod \"perf-node-gather-daemonset-5lpz6\" (UID: \"289b452d-7816-44a9-a028-d5d5844060c6\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" Nov 02 18:59:02.117774 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:02.117696 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/289b452d-7816-44a9-a028-d5d5844060c6-lib-modules\") pod \"perf-node-gather-daemonset-5lpz6\" (UID: \"289b452d-7816-44a9-a028-d5d5844060c6\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" Nov 02 18:59:02.117774 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:02.117742 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/289b452d-7816-44a9-a028-d5d5844060c6-podres\") pod \"perf-node-gather-daemonset-5lpz6\" (UID: \"289b452d-7816-44a9-a028-d5d5844060c6\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" Nov 02 18:59:02.117774 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:02.117689 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/289b452d-7816-44a9-a028-d5d5844060c6-sys\") pod \"perf-node-gather-daemonset-5lpz6\" (UID: \"289b452d-7816-44a9-a028-d5d5844060c6\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" Nov 02 18:59:02.125843 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:02.125817 5601 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hn6z\" (UniqueName: \"kubernetes.io/projected/289b452d-7816-44a9-a028-d5d5844060c6-kube-api-access-4hn6z\") pod \"perf-node-gather-daemonset-5lpz6\" (UID: \"289b452d-7816-44a9-a028-d5d5844060c6\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" Nov 02 18:59:02.201804 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:02.201770 5601 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" Nov 02 18:59:02.475719 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:02.475687 5601 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6"] Nov 02 18:59:02.478287 ip-10-0-141-169 kubenswrapper[5601]: W1102 18:59:02.478256 5601 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod289b452d_7816_44a9_a028_d5d5844060c6.slice/crio-c88414dfe42a0c3a52ced9cb30311efc4a8d11bbe9f86ed4691a85879b64307b WatchSource:0}: Error finding container c88414dfe42a0c3a52ced9cb30311efc4a8d11bbe9f86ed4691a85879b64307b: Status 404 returned error can't find the container with id c88414dfe42a0c3a52ced9cb30311efc4a8d11bbe9f86ed4691a85879b64307b Nov 02 18:59:03.496105 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:03.496069 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" event={"ID":"289b452d-7816-44a9-a028-d5d5844060c6","Type":"ContainerStarted","Data":"99f5bddd7dc85b0056197481db4e04ad8030d641e427fa5fd9d39c74aab572e1"} Nov 02 18:59:03.496523 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:03.496159 5601 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" event={"ID":"289b452d-7816-44a9-a028-d5d5844060c6","Type":"ContainerStarted","Data":"c88414dfe42a0c3a52ced9cb30311efc4a8d11bbe9f86ed4691a85879b64307b"} Nov 02 18:59:03.496523 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:03.496183 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" Nov 02 18:59:03.513020 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:03.512973 5601 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" podStartSLOduration=2.512961415 podStartE2EDuration="2.512961415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 18:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:59:03.510229882 +0000 UTC m=+5108.201283896" watchObservedRunningTime="2026-11-02 18:59:03.512961415 +0000 UTC m=+5108.204015468" Nov 02 18:59:04.483014 ip-10-0-141-169 kubenswrapper[5601]: E1102 18:59:04.482978 5601 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vrm5q" containerName="extractor" Nov 02 18:59:04.538738 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:04.538643 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-t5dg5_aad4749c-18d8-4003-8576-211672ff42e5/manager/0.log" Nov 02 18:59:05.417966 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:05.417928 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-hsptq_3aa1eac9-b956-4608-8c5a-343b37a69d70/manager/0.log" Nov 02 18:59:07.141690 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:07.141658 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rtxf5_afbca029-7d59-4cfb-b08c-353e174b0355/dns/0.log" Nov 02 18:59:07.167204 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:07.167176 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rtxf5_afbca029-7d59-4cfb-b08c-353e174b0355/kube-rbac-proxy/0.log" Nov 02 18:59:07.223766 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:07.223732 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5tkc8_a48715bf-e5aa-4b50-a9a5-4916e6fbefbc/dns-node-resolver/0.log" Nov 02 18:59:08.094645 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:08.094602 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q28kf_d9300fb6-6926-4429-8797-603dfb361818/node-ca/0.log" Nov 02 18:59:08.554650 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:08.554602 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-t5dg5_aad4749c-18d8-4003-8576-211672ff42e5/manager/0.log" Nov 02 18:59:09.468289 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:09.468255 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-hsptq_3aa1eac9-b956-4608-8c5a-343b37a69d70/manager/0.log" Nov 02 18:59:09.514604 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:09.514577 5601 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-5lpz6" Nov 02 18:59:09.667248 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:09.667216 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-dl596_f0e27933-7691-4793-922c-4e83bba3914e/discovery/0.log" Nov 02 18:59:10.049722 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:10.049642 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cf9f7b655-m7r89_9d1ee04f-2201-4da0-88ff-f06683fc6006/kube-auth-proxy/0.log" Nov 02 18:59:10.184042 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:10.184009 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-jv8f4_c32fa6f3-9665-4b3b-b093-22435190829b/istio-proxy/0.log" Nov 02 18:59:10.292452 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:10.292415 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-th4lt_0659edff-3cd9-495d-bbce-c19f48452bf4/istio-proxy/0.log" Nov 02 18:59:10.325751 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:10.325663 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-wtks5_63f70366-70f1-475a-8e45-eb6da6b4828e/istio-proxy/0.log" Nov 02 18:59:10.330644 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:10.330595 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-wtks5_63f70366-70f1-475a-8e45-eb6da6b4828e/istio-proxy/1.log" Nov 02 18:59:12.023512 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:12.023477 5601 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg"] Nov 02 18:59:12.026885 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:12.026849 5601 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e66f0f-zcsqg"] Nov 02 18:59:12.978194 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:12.978168 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-t5dg5_aad4749c-18d8-4003-8576-211672ff42e5/manager/0.log" Nov 02 18:59:13.425775 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:13.425731 5601 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf94ee92-6bf4-449f-b97c-80ee9cb0c41b" path="/var/lib/kubelet/pods/bf94ee92-6bf4-449f-b97c-80ee9cb0c41b/volumes" Nov 02 18:59:13.904265 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:13.904182 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-hsptq_3aa1eac9-b956-4608-8c5a-343b37a69d70/manager/0.log" Nov 02 18:59:14.604501 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:14.604475 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nwbhl_a47b32de-3a53-4b34-8874-87049c4d747b/serve-healthcheck-canary/0.log" Nov 02 18:59:15.378823 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:15.378787 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-dl596_f0e27933-7691-4793-922c-4e83bba3914e/discovery/0.log" Nov 02 18:59:15.391137 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:15.391102 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cf9f7b655-m7r89_9d1ee04f-2201-4da0-88ff-f06683fc6006/kube-auth-proxy/0.log" Nov 02 18:59:15.475701 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:15.475670 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-jv8f4_c32fa6f3-9665-4b3b-b093-22435190829b/istio-proxy/0.log" Nov 02 18:59:15.496917 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:15.496883 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vrm5q_9372be33-753d-4006-ba16-86fc44bd7c52/kube-rbac-proxy/0.log" Nov 02 18:59:15.520849 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:15.520822 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vrm5q_9372be33-753d-4006-ba16-86fc44bd7c52/exporter/0.log" Nov 02 18:59:15.531218 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:15.531190 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-th4lt_0659edff-3cd9-495d-bbce-c19f48452bf4/istio-proxy/0.log" Nov 02 18:59:15.546377 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:15.546357 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vrm5q_9372be33-753d-4006-ba16-86fc44bd7c52/extractor/0.log" Nov 02 18:59:15.551977 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:15.551956 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-wtks5_63f70366-70f1-475a-8e45-eb6da6b4828e/istio-proxy/0.log" Nov 02 18:59:15.555799 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:15.555768 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-wtks5_63f70366-70f1-475a-8e45-eb6da6b4828e/istio-proxy/1.log" Nov 02 18:59:17.491523 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:17.491494 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-dl596_f0e27933-7691-4793-922c-4e83bba3914e/discovery/0.log" Nov 02 18:59:17.502113 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:17.502083 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cf9f7b655-m7r89_9d1ee04f-2201-4da0-88ff-f06683fc6006/kube-auth-proxy/0.log" Nov 02 18:59:17.601896 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:17.601861 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-jv8f4_c32fa6f3-9665-4b3b-b093-22435190829b/istio-proxy/0.log" Nov 02 18:59:17.663105 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:17.663076 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-th4lt_0659edff-3cd9-495d-bbce-c19f48452bf4/istio-proxy/0.log" Nov 02 18:59:17.681406 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:17.681379 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-wtks5_63f70366-70f1-475a-8e45-eb6da6b4828e/istio-proxy/0.log" Nov 02 18:59:17.684438 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:17.684410 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-wtks5_63f70366-70f1-475a-8e45-eb6da6b4828e/istio-proxy/1.log" Nov 02 18:59:18.405361 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:18.405333 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-789486f7c8-xc65w_40b11c81-5baa-481c-b138-3ddc1f2e633c/manager/0.log" Nov 02 18:59:18.432153 ip-10-0-141-169 kubenswrapper[5601]: I1102 18:59:18.432128 5601 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-789486f7c8-xc65w_40b11c81-5baa-481c-b138-3ddc1f2e633c/copy-manifests/0.log"